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ABSTRACT 


The  first  one  hundred  days  of  combat  are  the  most  dangerous  time  for  a  soldier 
due  to  uncertainty  and  unfamiliarity  with  his  area  of  operations.  Developing  a 
desktop  virtual  trainer  for  change  detection  could  reduce  the  threat  to  soldiers  by 
improving  their  ability  to  detect  signals  among  the  cluttered,  noisy  neighborhoods 
of  the  current  operational  environment.  Building  upon  previous  change  detection 
and  signal  detection  work,  this  thesis  explores  the  use  of  Army  Virtual 
Battlespace  2™  as  a  prototype-training  tool  for  change  detection.  Leveraging  an 
Army-owned  distributable  trainer  would  potentially  benefit  soldiers  prior  to 
deployment.  This  research  team  conducted  an  experiment  that  tested  fifteen 
participants  over  four  weeks.  Each  participant  explored  the  virtual  environment 
twelve  times.  Researchers  analyzed  correct  detections,  false  alarms,  user 
confidence,  threat  levels,  and  tutorial  group  assignment.  This  first  attempt  at 
developing  a  military-oriented  virtual  trainer  resulted  in  statistically  significant 
improvement  in  detection  percentages,  user  confidence,  and  decay  of  false- 
alarm  rates  over  time  with  p-values  less  than  0.01.  The  results  showed  no 
significance  in  the  use  of  an  in-simulation  tutorial  or  target  threat  level.  Future 
work  should  expand  on  this  foundational  research  to  determine  whether  the  skills 
developed  using  this  trainer  transfer  to  real-world  change  detection. 
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I.  INTRODUCTION 


A.  PROBLEM  STATEMENT 

The  current  operational  environment’s  (COE)  unique  challenges  demand 
soldiers  acquire  a  new,  more  robust  skill  set.  In  an  asymmetric  environment,  the 
enemy  blends  into  the  population  and  uses  unconventional  weapons  and  tactics. 
Those  who  operate  frequently  in  a  counterinsurgency  (COIN)  environment 
exercise  many  skills  specific  to  an  asymmetric  environment.  Most  operational 
units  revisit  the  same  terrain  many  times,  sometimes  for  their  entire  rotation  in 
Iraq  or  Afghanistan.  Asymmetric  threats  highlight  the  importance  of  heightened 
awareness  and  sharp  memory  in  order  to  recognize  environmental  and  social 
changes.  If  these  skills  do  in  fact  mitigate  the  risk  of  attacks,  then  recognizing 
objects,  people  and  behaviors  that  differ  from  previous  patrols  through  an  area  is 
critical  to  mission  success  and  force  protection.  Performing  detection  tasks 
under  stressful  combat  conditions  necessitates  the  rapid  development  of  change 
detection  skills  prior  to  deployments. 

The  Army  considers  the  first  one  hundred  days  in  combat  the  most  risky  to 
soldiers  (Center  for  Army  Lessons  Learned,  2008).  Learning  the  environment 
and  becoming  sensitive  to  change  are  fundamental  for  combat  skills 
development  during  this  initial  period.  Currently,  some  training  designed  to  hone 
these  abilities  occurs  during  pre-deployment  training.  Normally,  the  training 
includes  a  mixture  of  classroom  briefings  and  Situational  Training  Exercise  (STX) 
lane  training.  STX  for  detection  tasks  revolve  around  counter-IED  operations. 
This  training  varies  slightly  by  operational  unit.  Additionally,  several  new  virtual 
counter-IED  trainers  such  as  the  Institute  for  Creative  Technology’s  Mobile 
Counter-IED  Interactive  Trainer  (MCIT)  address  change  recognition  skills  (Peck, 
2010).  This  existing  training  is  significant  but  it  focuses  almost  exclusively  on 
training  tactical  preparation  and  response  to  IED  attacks.  One  problem  with  this 
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solution  is  the  limited  distribution  of  the  existing  virtual  trainers,  so  not  all  units 
are  able  to  benefit  from  this  tool.  Throughput  is  another  problematic  issue. 
Limited  virtual  trainers  do  not  enable  adequate  throughput,  and  often  result  in 
only  one  training  session  for  each  deploying  soldier.  When  the  enemy  emplaces 
lEDs,  there  is  typically  an  impact  or  change  in  the  immediate  environment.  Such 
changes  might  include  differences  in  the  appearance  of  road  or  ground  features. 
Changes  in  street  activity  may  indicate  a  threat.  Objects  such  as  parked 
vehicles,  storefronts,  and  number  or  type  of  people  on  a  street  represent  possible 
indicators.  Sometimes  changes  might  be  more  subtle  and  difficult  to  detect. 
Veteran  soldiers  gain  experience  in  identifying  both  obvious  and  subtle  changes 
to  the  environment  that  may  serve  as  a  warning.  This  thesis  attempts  to 
determine  whether  a  soldier’s  change  detection  skills  improve  through  training 
prior  to  deployment. 

By  developing  a  desktop  virtual  training  system  with  the  specific  purpose 
of  improving  the  recognition  of  change,  a  viable,  focused,  and  easily  distributable 
training  package  becomes  readily  available  to  all  deploying  units.  Furthermore, 
this  potential  solution  provides  near  unlimited  use  for  soldiers,  commanders  and 
staffs  seeking  to  improve  change  detection  and  memory  skills  of  combat  forces. 

The  Department  of  Defense  commits  money,  time,  and  personnel  into 
virtual  technologies,  and  they  have  continued  this  trend  over  the  past  three 
decades.  This  allocation  of  resources  creates  opportunity  for  new  training 
approaches  to  old  problems.  The  development  of  a  soldier’s  situational 
awareness,  which  comes  in  many  forms,  is  just  one  example  of  how  the 
Department  of  Defense  can  leverage  the  use  of  virtual  environments.  Being  alert 
to  changes  in  an  environment  is  one  manifestation  of  an  individual’s  level  of 
situational  awareness.  A  soldier’s  ability  to  detect  change  in  their  area  of 
responsibility  (AOR)  is  an  important  skill  that  might  improve  with  the  use  of  a 
desktop  training  solution. 
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The  United  States  Army  and  Marine  Corps  employ  Virtual  Battle  Space  2 
(VBS2)  as  a  portable,  personal  virtual  training  environment  for  their  personnel. 
Given  the  limited  training  resources  of  the  armed  services,  VBS2  represents  a 
viable  way  to  train  ground  combat  tasks  applicable  to  the  COE.  The  use  of 
virtual  simulation  allows  multiple  training  iterations  at  very  little  cost  to  the 
government.  Although  virtual  environments  offer  many  opportunities,  the  current 
challenges  are  to  determine  (1)  what  tasks  to  train  in  VBS2,  (2)  how  to  measure 
training  progress  using  this  new  tool,  (3)  how  to  validate  positive  training  transfer 
on  these  tasks,  and  (4)  what  instructional  support  materials  might  be  needed  to 
provide  for  each  training  package. 

Threat  detection  or  change  detection  skills  are  one  example  of  a  common 
combat  task  that  might  improve  by  using  a  virtual  environment  for  training.  This 
thesis  seeks  to  investigate  the  potential  for  using  VBS2  or  a  similar  virtual 
environment  as  a  desktop  trainer  for  change  detection  skills. 

B.  RESEARCH  QUESTIONS 

1.  Does  change  recognition  performance  improve  over  time  in  a  virtual 
environment?  How  does  it  improve? 

a.  Can  a  participant  improve  at  recognizing  that  something  has 
changed  in  general,  measured  by  correct  detections  or  detection 
percentages? 

b.  Does  a  participant’s  confidence  level  improve  at  recognizing  and 
identifying  changes  in  a  scene  using  a  simple  ordinal  scale  of  high, 
medium,  and  low? 

2.  How  do  detection  percentages  differ  when  assessing  a  participant’s 
performance  on  a  variety  of  threat  difficulty  levels? 

3.  How  does  automated  virtual  training  affect  change  detection 
percentages? 
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4.  Does  the  percentage  of  recognized  changes  and  false  alarms  grow 
proportionally?  Alternatively,  can  the  detection  percentage  improve  while 
simultaneously  decreasing  the  false  alarm  rate  through  repeated  exposure  to  a 
simulated  environment? 

C.  HYPOTHESES  (GIVEN  TRAINING  OR  EXPOSURE  IN  VE) 

1 .  Participants’  detection  rates  will  increase  over  time. 

2.  Participants’  false  alarm  rates  will  decrease  over  time. 

3.  Participants’  confidence  will  increase  over  time  when  detecting 
changes  in  a  virtual  environment. 

4.  Detection  rates  will  be  higher  when  evaluated  against  a  high  threat 
level,  a  subjective  measure  of  how  threatening  a  scene  appears. 

5.  Participants  receiving  the  automated  virtual  training  will  detect  more 
changes  (perform  better)  in  signal  plus  noise  scenarios. 

D.  OTHER  EXPLORATORY  QUESTIONS 

Do  factors  such  as  age,  rank,  branch-of-service,  combat  experience,  or 
video  game  experience  affect  a  participant’s  ability  to  learn  to  detect  changes  in 
a  virtual  environment? 

Do  participants  believe  they  received  quality  training  that  could  positively 
influence  real-world  change  detection  performance? 

E.  SCOPE 

The  scope  of  this  thesis  focuses  on  answering  these  research  questions. 
The  research  team  developed  the  five  hypotheses  to  guide  the  experiment  and 
answer  the  research  questions.  This  research  endeavor  developed  a  prototype 
change-recognition  trainer  using  VBS2.  The  team  tested  the  trainer  with  a 
sample  population  from  Naval  Postgraduate  (NPS)  students  in  order  to  measure 
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the  training  effectiveness  of  the  virtual  environment  over  a  period  of  weeks 
through  multiple  exposures.  After  testing,  the  team  analyzed  the  data,  identified 
trends  and  explained  the  results.  Finally,  the  report  concludes  with 
recommended  future  work  and  an  exploration  of  potential  distribution  methods  for 
VBS2  change  detection  scenarios. 

F.  BENEFITS  OF  STUDY 

By  demonstrating  that  virtual  environment  training  improves  change 
detection  skills,  this  study  highlights  the  capability  of  VBS2  as  a  desktop  part-task 
trainer  for  soldiers.  Because  all  U.S.  Army  soldiers  have  access  to  VBS2  on  their 
personal  computers,  this  research  has  the  potential  to  make  change  detection 
training  more  accessible  to  the  individual  user.  Similar  training  could  reduce  the 
risk  of  casualties  from  a  variety  of  asymmetric  threats  in  the  first  one-hundred 
days  in  combat  and  beyond. 

G.  THESIS  ORGANIZATION  AND  TABLE  OF  CONTENTS 

•  Chapter  I:  Introduction.  This  chapter  describes  the  problem,  lists 
the  research  questions,  presents  the  hypothesis,  and  defines  the 
scope  and  benefits  of  this  study. 

•  Chapter  II:  Background.  This  chapter  provides  a  literature  review 
for  the  study.  This  review  includes  current  literature  on  computer 
games  for  training,  arousal,  vigilance,  memory,  attention,  change 
detection,  Signal  Detection  Theory,  and  Army  training  challenges. 

•  Chapter  III:  Methodology.  This  chapter  describes  how  the  research 
team  designed  the  experiment,  including  participants,  procedures, 
and  materials. 

•  Chapter  IV:  Results  and  Discussion.  This  chapter  contains  the 
results  of  experimentation  and  an  interpretation  of  those  results. 

•  Chapter  V:  Recommendations.  This  chapter  provides  an  overall 
assessment,  methods  for  using  VBS2  scenarios,  and  recommends 
future  work  for  change  detection  in  virtual  environments. 

•  Appendix  A:  Pre-Experiment  Instructions.  This  appendix  contains 
all  materials  each  participant  received  prior  to  the  experiment. 
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•  Appendix  B:  Computer  Set-up.  This  appendix  describes  how  to 
set-up  and  begin  change  detection  training  in  VBS2 

•  Appendix  C:  Approved  IRB  Protocol.  This  appendix  contains  the 
approved  Institutional  Review  Board  protocol  for  experimentation 
with  human  subjects. 

•  Appendix  D:  Informed  Consent.  This  appendix  shows  the  consent 
form  signed  by  all  participants. 

•  Appendix  E:  Demographic  Survey.  This  appendix  contains  the 
demographic  survey  completed  by  all  participants. 

•  Appendix  F:  VBS2  Scripts.  This  appendix  displays  all  scripts 
(code)  for  virtual  activity  in  the  scenes  and  human-computer 
interaction  with  the  simulation. 

•  Appendix  G:  Tutorial  Voice-Over  Scripts.  This  appendix  contains 
all  the  scripts  for  voice-overs  heard  in  VBS2  by  participants. 

•  Appendix  H:  JMP  Raw  Data.  This  appendix  displays  the  raw  data 
matrix  used  to  record  participant  data  during  experimentation. 

•  Appendix  I:  Confidence  vs.  Threat  Raw  Data  Matrix.  This  appendix 
shows  the  SME  threat-level  ratings  and  participant  confidence  raw 
data  table. 

•  Appendix  J:  How  to  create  a  Change  Detection  Scenario.  This 
appendix  expands  upon  Chapter  III,  providing  more  depth  on  how 
to  create  a  change  detection  scenario  in  VBS2.  Additionally,  it 
provides  the  reader  step-by-step  instructions  for  repeating  the 
experiment. 

•  Appendix  K:  Changes  Used  in  this  Experiment.  This  appendix 
shows  all  before  and  after  pictures  of  the  changes  used  in  this 
experiment  providing  additional  context. 

•  Appendix  L:  Potential  Target  (Change)  Locations.  This  appendix 
contains  pictures  of  the  entire  virtual  environment  used  in  this 
experiment.  It  shows  a  comprehensive  list  of  target  areas  possible 
in  the  virtual  environment. 
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II.  BACKGROUND 


A.  INTRODUCTION 

The  background  work  for  this  thesis  examined  several  disciplines  and 
knowledge  areas.  First,  the  research  team  examined  previous  research  that 
leveraged  video  game  and  simulation  technologies  for  training  ensured  that  the 
research  team  selected  a  viable  tool  for  training  change  detection.  Specifically, 
the  military’s  use  of  video  game-based  training  was  of  significant  importance.  A 
review  of  this  literature  revealed  possible  indicators  for  how  a  participant  might 
perform  in  a  virtual  environment.  Next,  the  team  sought  to  understand  the 
relationship  between  arousal,  vigilance,  attention,  and  memory  helped  guide  the 
development  of  an  engaging  and  interesting  scenario.  Proper  scenarios  were  of 
paramount  importance  in  order  to  stimulate  and  motivate  participants  during 
experimentation.  An  understanding  of  Signal  Detection  Theory  (SDT)  and 
previous  work  done  with  change  detection  provided  a  starting  point  for  continuing 
research  and  formed  the  basis  for  measuring  the  effects  of  this  experiment. 
Finally,  the  research  team  reviewed  at  studies  outlining  future  requirements  for 
Army  training. 

B.  USE  OF  VIDEO  GAMES  FOR  TRAINING 

Brehmer  and  Dorner  (1993)  use  the  term  “microworlds”  to  describe  video 
games  and  simulations.  The  authors  explain  that  the  problem  with  research  in  a 
lab  or  in  the  field  is  that  both  of  those  options  contain  great  advantages  and 
disadvantages.  In  a  lab,  the  research  team  can  control  almost  all  variability; 
however,  individuals  conducting  those  tasks  in  the  real  world  often  call  the 
external  validity,  or  generalizability,  of  the  lab  results  into  question.  On  the  other 
side  of  the  coin,  experimentation  in  the  field  contains  innumerable  confounds, 
due  to  a  lack  of  control.  While  user  communities  view  field  studies  as  valid, 


7 


scientists  often  question  the  conclusions.  The  use  of  games  or  simulations 
provides  a  balance  between  these  two  extremes.  Virtual  environments  allow  for 
more  control  than  the  field  and  more  complexity  than  the  lab  (Brehmer  &  Dorner, 
1993). 

By  identifying  the  variables  that  affect  training  processes  and  procedures 
early  on,  training  developers  can  structure  virtual  training  tools  to  maximize 
training  benefits.  A  popular  training  apparatus  today  is  the  computer  video  game 
for  individual  and  collective  training.  If  designed  properly,  games  represent  an 
entertaining  and  educational  medium  for  training  military  skills.  Experience, 
orientation  periods,  and  input  devices  are  important  factors  to  consider  when 
designing  a  game  for  training.  The  next  few  sections  address  the  concerns  in 
turn. 


1.  Video  Game  Experience 

In  a  study  conducted  by  Orvis,  Moore,  Belanich,  Murphy  and  Horn  (2010), 
they  found  that  soldiers  were  not  typically  “gamers.”  In  their  study,  Orvis  defined 
the  term  “gamer”  as  someone  who  plays  video  games  at  least  once  a  week. 
They  found  the  percentage  of  soldiers  who  reported  being  gamers  was  less  than 
43  percent  (Orvis,  Moore,  Belanich,  &  Horn,  2010). 

In  another  study  by  Orvis,  Horn  and  Belanich  (2009),  they  demonstrated 
that  prior  gaming  experience  increased  the  time  subjects  stayed  on  task.  In 
other  words,  it  took  longer  for  gamers  to  become  bored  from  the  training. 
Gamers  also  demonstrated  increased  training  performance  and  greater  overall 
satisfaction  with  the  training. 

A  study  by  Green  and  Bavelier  (2003)  on  visual  selective  attention  for 
gamers  and  Non-Video  Game  Players  (NVGPs)  showed  that  gaming  skills  are  a 
learned  task.  Across  multiple  experiments,  gamers  demonstrated  greater 
attention  capacity  at  visual  tasks  than  their  NVGP  counterparts.  Green  and 
Bavelier  feared  that  NVGPs’  visual  abilities  might  not  be  as  good  as  the  gamers, 
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and  thus,  they  did  not  play  video  games  as  a  result.  To  eliminate  this  possible 
confound,  Green  and  Bavelier  split  the  NVGPs  into  two  groups  that  practiced 
playing  video  games.  For  this  experiment,  one  group  practiced  with  the  action 
game  Medal  of  Honor,  a  game  similar  to  the  games  reportedly  played  by  the 
Video  Game  Players  (VGP)  in  previous  experiments.  The  name  given  to  this 
group  was  “action  game.”  The  “control  game”  group  practiced  on  Tetris.  Since 
Tetris  is  a  monofocus  game  vice  the  multifocus  Medal  of  Honor,  the  research 
team  did  not  expect  to  see  altered  attention  ability  in  the  control  game  group. 
After  one  hour  of  practice  a  day  for  ten  days,  the  action  game  group  showed 
improvement  in  video  game  playing.  The  Tetris  or  control  group,  showed  no 
improvement  (Green  &  Bavelier,  2003). 

In  Green  and  Bavelier’s  experiment,  the  gamers  showed  the  highest 
attentional  capacity  of  all  participants.  However,  the  improvement  of  the  NVGP 
group  playing  action  video  games  demonstrated  that  playing  complex  video 
games  could  alter  visual  attention  by  speeding  up  the  perceptual  processes 
required  by  the  games  (Green  &  Bavelier,  2003). 

2.  Orientation  Periods 

Pretraining  is  any  training  that  occurs  before  an  actual  training  exercise. 
Pre-Marksmanship  Instruction  (PMI)  is  a  good  military  example  of  pretraining. 
PMI  is  static  weapons  training  conducted  by  military  forces,  often  in  garrison, 
before  going  to  a  live  weapons  range.  Building  pretraining  into  a  training 
program  of  instruction  (POI)  fosters  improved  performance  and  increases  the 
value  of  training  in  a  virtual  environment.  Orientation  periods,  self-paced 
practice,  and  instructor  set  training  scenarios  allow  for  personalized  training 
(Orvis  et  al . ,  2010).  Not  all  soldiers  have  the  same  level  of  experience  using 
simulations  and  video  games.  Acknowledging  this  fact  and  building  the  training 


9 


system  to  accommodate  these  differences  allows  the  more  experienced  players 
to  progress  faster  and  quickly  get  novice  players  to  an  acceptable  proficiency 
level. 

Flexibility  in  pretraining  sessions  decreases  both  boredom  from  training  for 
gamers  and  frustration  of  the  training  for  NVGPs.  When  learning  a  new  task, 
length  of  time  and  the  processes  necessary  for  skill  acquisition  are  not  clearly 
identifiable.  However,  “it  is  commonly  agreed  upon  that  time  spent  on  a  task  is 
the  prominent  cause  of  whether  or  not  the  skill  will  be  acquired”  (Vowels,  2010,  p. 
17).  Variation  in  pretraining  length  or  difficulty  allows  those  requiring  more  time 
or  focus  on  a  specific  area  to  receive  it.  Individualized  pretraining  variability 
improves  the  overall  training  experience  while  standardizing  the  training. 

3.  Ease  of  Use  and  Familiarity  with  Input  Devices 

Identifying  the  appropriate  input  device  for  use  in  virtual  environment 
training  scenario  is  critical.  There  are  measurable  differences  between  various 
input  devices.  Research  on  input  devices  among  young  adults  indicated  that 
“some  devices  are  better  than  others;  no  one  device  has  been  shown  to  be 
superior  for  all  tasks  or  applications  among  young  adult  populations”  (Wood, 
Willoughby,  Rushing,  Bechtel,  &  Gilbert,  2005).  Identifying  the  input  device  that 
allows  a  user  to  quickly  reach  asymptotic  performance  could  lead  to  less  pre¬ 
training  requirements,  thus  allowing  trainees  to  focus  on  the  critical  tasks  taught 
by  a  virtual  trainer. 

Ease  of  use  and  comfort  level  with  the  input  device  affect  performance 
levels.  If  trainees  find  the  interface  and  “learning  environment  to  be  frustrating 
and  difficult  to  use,  they  may  experience  decreased  motivation  that  prevents 
them  from  fully  engaging  in  or  completing  the  instruction”(Orvis,  et  al.  2009). 
Along  with  potential  interface  frustrations,  vigilance  decrement  also  effects 
attention  levels.  As  outlined  by  Proctor  and  Van  Zandt  (2008),  task  vigilance 
decreases  dramatically  after  thirty  minutes.  Therefore,  it  is  important  that  the 


10 


trainee  continue  to  feel  engaged,  motivated,  and  focused  during  training 
sessions.  The  input  device  has  a  profound  impact  on  this  feeling  of  engagment 
or  frustration,  and  thus,  it  is  a  valuable  part  of  the  training  experience. 
Consideration  and  careful  evaluation  must  go  into  the  selection  of  an  input  device 
for  any  training  envrionment. 

C.  AROUSAL 

After  gaining  a  trainee’s  interest,  maintaining  that  attention  is  essential  for 
a  successful  training  tool.  The  first  step  to  draw  the  interest  in  the  tool  is  through 
what  Jane  Mackworth  (1968)  calls  the  arousal  response.  The  arousal  response 
is  a  neural  reaction  to  a  stimulus  alerting  the  brain.  Mackworth  argues  that  high 
motivation  and  knowledge  of  results  (KR)  are  key  factors  to  maintaining  the  high 
arousal  necessary  to  staying  alert  at  a  vigilance  task  such  as  signal  or  change 
detection.  Researchers  employ  a  number  of  techniques  in  order  to  maintain 
arousal,  motivation,  and  focused  attention  during  a  training  task.  These 
techniques  include  providing  the  participant  with  KR,  limiting  the  exposure,  and 
being  deliberate  about  the  number  of  signals  presented. 

D.  VIGILANCE 

Significant  research  on  vigilance  and  the  vigilance  decrement  followed 
World  War  II  where  many  inventions,  like  radar  and  sonar,  required  a  high  level 
of  operator  vigilance  in  order  to  accomplish  critical  military  tasks.  Since  that  time, 
our  reliance  on  technology  to  advance  military  capabilities  has  increased 
steadily.  This  section  explores  a  few  pioneers  in  vigilance  research  and  modern 
developments  to  an  age-old  problem. 

1.  Vigilance  Research:  Formative  Years 

Norman  H.  Mackworth  and  Jane  F.  Mackworth  explored  issues 
surrounding  vigilance  throughout  the  1 950s  and  1 960s.  Perhaps  one  of  the  most 
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enduring  works  on  vigilance  was  a  1961  experiment  in  which  N.H.  Mackworth 
(1961)  used  the  now  famous  Jump  Clock  Test.  Mackworth’s  research  was  a 
response  to  the  British  government’s  request  for  experimentation  on  ways  to 
alleviate  the  strain  on  radar  and  antisubmarine  watchmen. 

To  test  the  vigilance  of  individuals,  Mackworth  had  his  participants,  service 
members,  sit  in  a  cabin  and  watch  a  clock-like  device.  The  device  resembled  a 
wall  clock  that  had  only  one  hand,  six  (6)  inches  long,  and  a  plain  white  face 
without  the  normal  clock  markings  for  minutes  and  hours.  He  informed 
participants  that  the  hand  would  move  around  the  face  of  the  clock,  and 
Mackworth  instructed  participants  to  watch  for  a  “double  movement”  (Mackworth, 
1961).  During  this  movement,  the  hand  advanced  twice  the  normal  distance,  or 
about  the  equivalence  of  two  minutes.  Participants  pressed  a  button  when  they 
recognized  this  change. 

The  clock  experiment  lasted  two  hours  for  each  subject.  During  the  two 
hours,  each  participant  conducted  four  20-minute  sessions  watching  at  the  clock, 
and  if  they  saw  the  extra  movement,  they  had  eight  (8)  seconds  to  respond.  If  a 
participant  failed  to  respond  within  eight  seconds,  it  counted  as  a  miss 
(Mackworth,  1961). 

Another  key  component  of  Mackworth’s  experiment  was  the  lack  of 
feedback  each  participant  received  during  each  trial.  This  feedback  was  an 
example  of  knowledge  of  results  (KR).  The  feedback  provided  to  the  participant 
could  have  come  during  the  experiment  or  between  trials  in  order  to  provide 
some  tangible  information  about  their  performance;  however,  Mackworth  chose 
to  provide  neither  to  his  subjects. 

To  summarize  Mackworth’s  extensive  experimentation,  he  found  that  over 
time,  participants  missed  more  and  more  signals.  He  also  experimented  with 
Benzedrine,  an  amphetamine,  and  found  that  five  of  every  six  participants  taking 
the  drug  performed  better  in  the  long  run;  however,  he  also  noted  that 
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performance  under  the  influence  of  Benzedrine  was  no  better  than  the  average 
participant  without  the  drug  in  the  first  30  minutes  of  the  test  (Mackworth,  1961). 

N.H.  Mackworth  made  two  important  conclusions  that  led  to  an  entire  field 
of  follow-on  research.  First,  thirty  minutes  seemed  to  be  a  significant  measure 
for  vigilance.  Essentially,  he  discovered  and  documented  the  vigilance 
decrement.  Second,  he  documented  the  fact  that  amphetamines  helped 
participants  perform  better  at  a  vigilance  task  for  longer  periods;  however,  the 
use  of  drugs  lost  acceptance  in  the  military  making  other  methods  necessary. 

2.  Defining  the  Vigilance  Decrement 

Jane  Mackworth  wrote  a  paper  for  Psychological  Review  in  1968  that 
outlined  much  of  the  work  conducted  by  Norman  H.  Mackworth  and  other  experts 
on  vigilance.  She  elaborated  on  the  concept  of  habituation  as  it  related  to  a 
decrease  in  vigilance.  Specifically,  Jane  Mackworth  (1968)  argued  that  there 
were  two  factors  at  work  in  the  brain  causing  a  vigilance  decrement:  (1)  arousal 
response  and  (2)  evoked  potentials.  She  claimed  that  due  to  habituation  the 
arousal  response,  or  brain  rhythm,  becomes  less  sensitive  to  changes,  especially 
when  an  environment  is  noisy.  Conversely,  she  explained  that  habituation  in  the 
evoked  potentials,  produced  by  a  repetitive  stimulus  in  the  background  noise, 
cause  a  decrease  in  detections,  most  likely  when  the  event  rate  is  regular  or  slow 
(Mackworth,  1968). 

Jane  Mackworth’s  further  explanation  of  the  vigilance  decrement  is 
important,  and  today,  the  Mackworths’  concept  is  universally  accepted.  Jane 
Mackworth’s  work  is  significant  because  it  explains  that  there  is  more  than  just 
boredom  involved  when  observing  a  vigilance  decrement.  In  fact,  it  might  be  due 
to  overstimulation. 


13 


3.  Vigilance  at  Repetitive,  Boring  Tasks 


Robert  R.  Mackie  researched  vigilance  throughout  the  1970’s  through  the 
1990’s.  Another  military-related  research  project  involving  vigilance  was  Robert 
R.  Mackie,  C.  Dennis  Wylie,  and  Malcolm  J.  Smith’s  experiment  investigating 
sonar  watch.  Mackie  et  al.  described  three  characteristics  that  led  to  a  vigilance 
decrement:  (1)  boring  task,  (2)  low  signal  rate,  and  (3)  no  feedback  loop  (Mackie, 
Wylie,  &  Smith,  1994). 

During  watchstanding,  the  likelihood  of  seeing  a  significant  signal,  or 
threat  indicator,  is  often  very  low,  if  not  unlikely  all  together.  This  watchstanding 
phenomenon  has  similarities  to  tasks  required  of  soldiers  in  the  common 
operating  environments  of  Iraq  and  Afghanistan.  Daily  patrols  move  in  and  out  of 
neighborhoods,  often  without  experiencing  any  threatening  signals.  So  how  does 
low  signal  rate  affect  vigilance  and  what  training  is  necessary  to  maintain 
vigilance  under  these  circumstances?  Those  questions  formed  the  foundation  of 
Mackie’s  research. 

Mackie  et  al.  described  a  problem  with  experimentation  prior  to  their 
research.  They  believed  that  signals  in  vigilance  research  were  far  too  frequent 
to  replicate  reality  and  thus  have  validity  outside  the  lab.  They  acknowledged  the 
importance  of  knowledge  of  results  (KR),  but  he  noticed  that  there  was  a  dearth 
of  research  into  artificial  signal  injection  aimed  at  improving  motivation  (Mackie  et 
al.,  1994).  Their  1994  study  sought  to  determine  how  signal  injection  and 
knowledge  of  results  would  affect  a  boring  task  like  sonar  watch. 

For  this  experiment,  Mackie  et  al.  set  up  a  computerized  sonar  display, 
closely  resembling  the  actual  equipment  used  by  the  British  Navy.  Each 
participant  sat  at  the  display  and  observed  the  signals  on  the  sonar  equipment. 
Normal  signals  looked  as  they  do  in  the  real  world.  Injected  signals  displayed  an 
additional  asterisk  on  the  signal  line  after  detected  by  a  participant.  Furthermore, 
an  alarm  sound  notified  a  participant  when  they  had  neglected  to  mark  an 
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injected  signal  after  it  had  been  present  for  5-minutes.  Participants  moved  a 
cursor  on  screen  and  pressed  a  “report”  key  when  detecting  a  real  or  injected 
signal  (Mackie  et  al.,  1994). 

There  are  numerous  lessons  to  learn  from  this  research  team’s 
methodology.  First,  they  are  purposefully  keeping  participants  engaged  in  the 
scenario  by  injecting  bogus  signals  that  look  like  the  real  world  threat  signals. 
Once  a  user  detects  an  injected  signal,  they  are  rewarded  with  knowledge  of 
results  (an  asterisk),  and  if  they  fail  to  catch  the  injected  signal,  they  are  alerted 
with  an  audible  tone  -  knowledge  of  results.  In  effect,  the  researchers  create  the 
conditions  necessary  to  actively  motivate  their  participants  and  keep  them  alert 
for  a  specified  signal  in  an  attempt  to  circumvent  the  vigilance  decrement. 

Mackies  et  al.  conducted  four  separate  experiments  as  part  of  this  study. 
Participants  performed  the  signal  detection  task  twice  for  a  length  of  2-hours  to 
5.5-hours  per  session,  depending  on  the  experiment  trial  number.  This  was  a 
long  experimentation  time  per  subject,  but  it  adds  to  the  validity  of  his  results 
since  tasks  like  these  are  normally  conducted  during  an  8-12  hour  shift.  It  is 
important  to  note  that  injecting  signals  produced  statistically  significant 
improvement  to  signal  detection,  especially  over  time  (Mackie  et  al.,  1994).  This 
constant  detection  forced  a  user  to  maintain  a  high  level  of  vigilance.  This 
focused  attention  prevented,  or  at  least  delayed,  the  vigilance  decrement. 

4.  Mental  Workload  and  Stress 

Where  the  Mackworths  and  Mackie  focused  on  boring,  mundane  tasks 
involving  detecting  a  short-duration,  infrequent  signal  in  a  sea  of  noise,  Joel  S. 
Warm  and  Raja  Parasuraman  (2008)  were  interested  in  high  mental  workload 
and  its  effect  on  vigilance.  Parasuraman’s  research  interests  included  human 
attention,  and  Warm’s  work  on  human  performance  in  military  systems  was 
noteworthy.  Warm,  Parasuraman,  and  Gerald  Matthews  authored  a  paper 
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outlining  a  new  approach  to  vigilance  research.  Their  approach  suggests  that, 
“vigilance  requires  hard  mental  work  and  is  stressful”  (Warm,  Parasuraman,  & 
Matthews,  2008,  433). 

To  support  this  argument,  Warm  et  al.  proclaimed  that  the  most  important 
finding  in  all  previous  vigilance  research  has  been  the  vigilance  decrement. 
However,  they  argue  that  previous  conclusions  might  be  incorrect  about  why  the 
decrement  occurs.  This  team  submits  that  recent  research  supports  the 
condition  where  a  vigilance  decrement  occurs  due  to  taxing  mental  requirements 
on  the  individual  conducting  a  vigilance  task.  They  draw  evidence  to  bolster  this 
argument  from  research  evaluating:  (1)  task  type,  (2)  perceived  mental  workload, 
(3)  neural  measures  of  resource  demand,  and  (4)  stress  related  to  the  task 
(Warm  et  al.,  2008). 

This  thesis  addresses  three  (3)  of  these  areas,  omitting  only  the  neural 
measures  of  resource  demand.  Although  the  objective  neural  measures  are 
important,  the  focus  of  this  research  was  to  identify  vigilance  considerations  for 
the  development  of  desktop  virtual  trainers.  The  average  developer  using  an 
application  like  VBS2  to  create  training  scenarios  does  not  have  access  to 
positron  emission  tomography  (PET)  or  functional  magnetic  resonance  imaging 
(fMRI)  machines  to  test  their  work. 

a.  Type  of  Task 

One  important  distinction  made  by  Warm  et  al.  is  the  type  of  task 
conducted  while  assessing  vigilance.  They  make  the  distinction  between 
successive  and  simultaneous  tasks.  Interestingly,  they  cite  research  supporting 
the  fact  that  successive  tasks  are  more  demanding  than  simultaneous  ones,  due 
to  the  requirement  to  use  memory  in  successive  tasks  (Warm  et  al.,  2008). 
Since  a  successive  task  scenario  would  be  more  demanding,  the  vigilance 
decrement  would  occur  sooner  during  this  type  set-up.  The  development  of  task 
objectives  requires  great  care  and  attention  to  detail.  The  sequence  or 
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presentation  of  signals  in  an  environment  can  affect  the  level  of  vigilance 
required  to  complete  any  task  or  training  event. 


b.  Perceived  Mental  Workload 

The  next  major  section  of  Warm’s  paper  deals  with  the  mental 
workload  on  the  individual  performing  a  task.  The  authors  describe  three 
components  of  this  workload:  (1)  mental,  (2)  physical,  and  (3)  temporal  (Warm  et 
al. ,  2008).  They  argue  that  a  finite  number  of  resources  exist,  and  once  those 
resources  are  gone,  a  vigilance  decrement  is  likely  to  follow.  Warm  et  al.  explain 
that  the  relationship  between  the  vigilance  decrement  and  workload  is  linear. 

This  argument  in  Warm’s  paper  has  significant  implications  for 
development  of  change  detection  scenarios  in  a  virtual  environment. 
Overloading  a  user  with  information  amid  the  requirement  to  detect  anomalies  in 
their  surroundings  quite  possibly  results  in  a  sharp  decrease  in  vigilance.  This 
decrement  will  lead  to  a  failure  of  the  training  system  to  achieve  its  intended 
purpose.  Showing  a  participant  where  to  look  in  their  environment  can  prevent 
such  an  overload,  and  explaining  what  to  look  for  should  lead  to  improved 
results. 


c.  User  Stress 

The  final  topic  discussed  in  Warm’s  paper  is  task-induced  stress. 
The  authors  acknowledge  that  using  objective  and  subjective  measures  to 
evaluate  stress  during  a  vigilance  task  adds  validity  to  the  evidence.  The  focus 
of  this  thesis  is  on  the  subjective  evaluations,  since  again,  most  military 
developers  of  desktop  solutions  do  not  have  access  to  laboratory  equipment  to 
measure  stress  objectively  in  their  trainees. 

Warm  and  his  colleagues  reported  that  participants  in  vigilance 
experiments  often  report  being  more  tired  and  stressed  after  the  exposure  than 
before  they  began  the  experiment  (Warm  et  al.,  2008).  They  compared  these 
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reports  to  the  amount  of  workload  required  during  the  experiment  and 
determined  that  stress  resulted  from  high  mental  workloads  during  the  testing. 

5.  Workload  Experiments 

Tiwari,  Singh  and  Singh  (2009)  also  studied  the  effects  of  task  demand 
and  workload  on  vigilance.  Much  like  Mackworth,  Tiwari  et  al.  (2009)  tested 
subjects  on  their  ability  to  detect  small  changes  in  a  common  object.  They  used 
a  square  that  would  occasionally  be  slightly  bigger  (a  half  centimeter,  3-3.5  cm) 
on  a  computer  screen.  They  tested  40  participants,  20  in  a  low-demand  task  (15 
events  per  minute)  and  20  in  a  high-demand  task  (thirty  events  per  minute).  The 
experiment  lasted  thirty  minutes  broken  into  three  (3)  10-minute  scenarios; 
afterwards,  participants  filled  out  a  questionnaire  to  ascertain  stress  and 
motivation  levels. 

Tiwari’s  research  team  affirmed  what  Warm  et  al.  said  about  workload  and 
stress.  There  were  many  interesting  subjective  results  from  the  questionnaires  to 
include  statistically  significant  changes  in  arousal,  motivation,  concentration,  and 
attention  levels.  Additionally,  those  participants  in  the  high-demand  task 
condition  rated  their  mental  workload  higher  (Tiwari  et  al.,  2009). 

On  the  objective  side,  participants  in  the  high-demand  group  detected 
fewer  correct  signals  and  committed  more  errors  than  those  in  the  low-demand 
group.  This  further  supported  the  idea  that  there  are  a  limited  number  of 
resources  to  draw  from  when  conducting  a  task  requiring  vigilance.  Overloading 
a  participant  appears  to  bring  about  a  vigilance  decrement,  just  as  easily  as  a 
monotonous  task  would.  Tiwari’s  team  confirmed  Warm  and  Parasuraman’s 
work  involving  mental  workload,  stress  and  vigilance.  It  also  supports  key 
considerations  for  future  development  with  virtual  environments  discussed  in  the 
methodology  section  of  this  thesis. 
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6.  Vigilance  Decrement  in  Virtual  Environments 


The  vigilance  decrement  is  a  potentially  devastating  phenomenon  that 
could  easily  devalue  any  instance  of  virtual  environment  training  where  it 
presents  itself.  As  military  professionals  tasked  with  developing  training  plans  for 
deploying  units,  this  research  team  must  remain  alert  to  vigilance  decrements 
and  understand  why  they  occur. 

Both  authors  have  seen  the  vigilance  decrement  many  times  through 
experience  with  military  training;  however,  until  recently,  explained  away  the 
phenomenon  as  boredom.  As  officers  who  have  been  through  boring, 
monotonous  training,  we  attributed  a  dip  in  trainee  performance  to  bad  design  of 
the  training  package.  This  attribution  oversimplified  the  problem. 

The  vigilance  decrement  explains  why  dips  in  performance  occur  during 
training.  Understanding  the  science  of  the  decrement  reveals  methods  to 
improve  the  training.  Using  previous  work  as  a  starting  point  and  heeding  the 
warnings  of  vigilance  researchers,  developers  can  accomplish  their  training  goals 
and  optimally  engage  the  training  audience  in  the  process. 

As  the  Mackworths  introduced  and  many  have  expanded  upon,  the 
vigilance  decrement  can  occur  because  of  over  stimulation  or  under  stimulation. 
Warm  and  Tiwari  confirmed  that  task  overload  problems  can  lead  to  stress  and 
cause  rapid  onset  of  the  vigilance  decrement.  The  rest  of  this  section  will 
describe  an  experiment  conducted  in  November  and  December  2010  where  this 
research  team  saw  a  vigilance  decrement  influence  the  results. 

7.  Vigilance  Decrement:  Thesis  Pilot  Study 

The  first  time  the  research  team  identified  a  vigilance  decrement  and  tried 
to  properly  classify  the  phenomenon  was  in  a  pilot  study  for  this  thesis  during  a 
Human  Factors  in  Systems  Design  class  at  the  Naval  Postgraduate  School 
(NPS).  The  research  team  consisted  of  Charalampos  Tsamtsaridis,  Gina 
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Becker,  Mike  Stinchfield,  and  Jason  Caldwell.  The  group  chose  to  pursue  a  topic 
close  to  this  thesis  area.  We  were  looking  at  input  devices  for  Virtual  Battle 
Space  2™  (VBS2)  to  evaluate  the  usability  of  each  device  for  thesis 
experimentation  (Tsamtsaridis,  Caldwell,  Stinchfield,  &  Becker,  2010). 

For  this  pilot  study,  the  team  developed  a  series  of  six  (6)  eight-minute 
scenarios  containing  60  potential  targets  in  the  virtual  world  of  VBS2.  Each 
simulation  scenario  had  three  (3)  distinct  areas  that  participants  traveled  through: 
the  first  contained  10  targets,  the  second  contained  20  targets,  and  the  third  had 
30  targets.  Participants  would  attempt  to  detect  targets  on  the  left  and  right  side 
of  a  narrow  road  while  riding  through  the  environment  as  a  passenger  in  the 
simulation  on  a  motorcycle.  The  team  conducted  the  pilot  study  over  two 
separate  sessions  where  participants  used  the  mouse  in  one  session  and  the 
head  tracker  in  the  other  session.  The  research  team  randomly  divided 
participants  between  group.  One  group  started  with  head  tracking  and  the  other 
started  with  the  mouse.  The  entire  experiment  lasted  approximately  two  (2) 
hours,  one  (1)  hour  for  each  session. 

The  research  goal  was  to  determine  the  time  to  reach  asymptotic 
performance  with  two  input  devices.  The  research  team  defined  asymptotic 
performance  as  a  less  than  10%  increase  in  targets  detected  for  three 
consecutive  scenarios.  While  plotting  the  data,  an  obvious  dip  in  performance 
occurred  over  the  course  of  experimentation.  This  performance  decrease 
showed  up  consistently  after  the  first  couple  of  scenarios  (Figure  1 ). 
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Targets  Identified  (%)  per  Repetition 
Using  the  Mouse  in  VBS2 


Figure  1 .  Vigilance  decrement  in  target  detection  task  with  the  mouse  as  the 

input  device 


At  the  time,  the  team  had  not  conducted  an  extensive  literature  review  on 
vigilance.  As  we  look  at  the  data  and  refer  back  to  Warm  and  Parasuraman’s 
explanation  for  the  vigilance  decrement,  our  pilot  study  team  was  guilty  of 
overloading  participants  and  tiring  them  out  very  quickly  by  immersing  them  in  a 
high-demand  task  over  an  extended  duration.  Task  overload  and  a  scenario 
length  exceeding  thirty  minutes  contributed  to  the  vigilance  decrement.  Subjects 
quickly  reached  asymptotic  performance  with  the  mouse  as  an  input  device  even 
with  a  vigilance  decrement,  but  using  the  TrackIR®  head  tracker  created  even 
more  problems. 

In  addition  to  vigilance  decrement  issues,  learning  to  use  the  TrackIR 
head  tracker  to  perform  a  detection  task  obviously  overwhelmed  the  participants. 
In  fact,  only  four  of  the  ten  participants  ever  reached  asymptotic  performance 
with  the  head  tracker  during  the  six  (6)  scenarios  involved  in  this  pilot  study. 
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Figure  2.  Vigilance  decrement  in  target  detection  task  with  the  head  tracker  as 

the  input  device 


Figure  2  shows  a  clear  decrement  after  the  second  scenario.  Additionally, 
it  illustrates  the  fact  that  six  of  our  ten  participants  never  reached  asymptotic 
performance  and  had  to  stop  the  experiment  after  the  designated  time. 

These  results  were  informative  and  necessitated  exploration  of  vigilance 
and  its  effects  prior  to  continuing  thesis  work.  Vigilance  greatly  affected  this 
research  team’s  approach  to  scenario  development  and  experimental  design 
using  VBS2.  Understanding  the  variables  that  change  a  user’s  vigilance  forced 
critical  thinking  about  the  way  to  evaluate  any  simulation  training. 

E.  ATTENTION 

Training  soldiers  and  marines  to  detect  threat  indicators  early  and 
accurately  can  save  lives  on  the  battlefield.  Arousal  and  vigilance  are  critical 
components  of  detection  skills  that  affect  overall  performance  search  tasks.  The 
Yerkes-Dodson  law  (Yerkes  &  Dodson,  1908)  says  that  a  person’s  best 
performance  can  be  estimated  based  on  arousal  level  and  task  complexity. 
Arousal  and  attention  are  closely  intertwined.  Both  relate  to  vigilance,  which 
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Proctor  and  Van  Zandt  define  simply  as  “sustained  attention,”  a  necessary 
component  for  detecting  change  (Proctor  &  Van  Zandt,  2008,  p.  244). 

Belopolsky,  Kramer,  and  Godign  (2008)  describe  how  selective  attention 
allows  humans  to  take  a  large  amount  of  visual  input  and  encode  that  into 
memory.in  both  top-down  and  bottom-up  methods.  They  argue  that  the  longer 
an  individual  fixates  on  an  object  in  a  scene,  the  more  likely  they  are  to  transfer 
that  to  memory.  Morelli  and  Burton  support  these  selective  attention  findings  and 
extend  the  work  to  the  military  domain  by  arguing  that  the  ability  to,  “filter  out 
distracting  information  and  selectively  attend  to  relevant  information  is  critical  to 
effective  performance  on  the  battlefield  (Morelli  &  Burton,  2009,  p.  81). 

Of  additional  concern  to  this  research  is  an  increased  attention  capacity  of 
video  game  players  (Green  &  Bavelier  2003).  Identifying  video  game  players 
becomes  increasingly  important  due  to  Green  and  Bavelier’s  result.  The 
research  team  acknowledges  that  VGP  might  actually  perform  better  at  change 
detection  in  a  virtual  environment  because  of  more  attention  resources.  Blocking 
by  this  criterion  is  important  to  the  validity  of  any  resulting  experimental  outcome. 

F.  MEMORY 

Once  a  trainee  employs  focused  or  selective  attention,  the  conditions  are 
set  to  encode  the  environment  into  memory.  Wickens  and  Hollands  (2000) 
describe  two  types  of  memory,  working  and  long-term.  They  purport  that  working 
memory  is  a  temporary  storage  area  until  it  can  be  stored  in  long-term  memory 
for  future  use. 

Wickens  et  al.  delineate  three  stages  of  memory:  encoding,  storage,  and 
retrieval.  For  the  purpose  of  this  thesis,  Wickens’  description  of  encoding  refers 
to  the  process  of  “learning  or  training”  used  to  move  spatial  information  from 
working  into  long-term  memory  (Wickens  &  Hollands,  2000,  p.242).  There  are 
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capacity  limitations  to  working  memory,  so  encoding  information  about  an  entire 
neighborhood  or  area  of  operations  is  problematic  and  most  likely  goes  into  long¬ 
term  episodic  memory  for  later  recall. 

Chunking  represents  one  popular  technique  of  grouping  smaller  pieces  of 
information  into  larger  “chunks”  for  encoding.  While  chunking  normally  refers  to 
techniques  for  managing  nominal  pieces  of  information,  it  is  reasonable  to 
believe  that  soldiers  unwittingly  employ  the  same  techniques  while  patrolling  in 
the  current  operating  environment.  Wickens  and  Hollands  propose  the  idea  of 
“skilled  memory”  derived  from  an  expert’s  ability  to  chunk  key  pieces  of 
information  (Wickens  &  Hollands,  2000,  p.  256).  It  is  easy  to  see  how  skilled 
memory  would  promote  the  detection  of  changes  in  a  soldier’s  environment. 


Figure  3.  Memory  functions  (From  Wickens  &  Hollands,  2000) 

After  encoding,  Wickens  and  Holland  explain  that  information  goes 
through  a  “storage”  stage.  They  explain  that  working  memory  uses  spatial  and 
verbal  cues,  while  long-term  memory  relies  on  declarative  and  procedural  models 
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(Wickens  &  Hollands,  2000,  p.  242).  While  important  concepts  to  understand, 
measuring  encoding  and  storage  is  problematic  for  someone  tasked  to  train 
soldiers  on  change  detection  tasks. 

From  a  training  evaluation  standpoint,  retrieval  represents  the  most 
important  and  measurable  stage  described  by  Wickens  and  Holland.  It  is  easy  to 
assess  retrieval,  since  a  soldier  either  remembers  or  forgets  details  about  their 
environment.  Wickens  and  Holland  describe  forgetting  as  a  “retrieval  failure” 
(Wickens  &  Hollands,  2000,  p.  242).  He  points  out  that  just  because  information 
is  in  long-term  memory  does  not  ensure  someone  will  remember  it.  He 
elaborates  on  two  types  of  memory  retrieval:  recall  and  recognition.  Recall  deals 
with  remembering  exact  information  in  memory.  Recognition  describes  the  ability 
to  answer  a  yes-or-no-type  question  accurately  based  on  information  stored  in 
memory  (Wickens  &  Hollands,  2000,  p.281).  For  a  change  detection  task, 
environmental  recognition  is  enough  to  enable  a  trainee  to  decide  if  a  change 
has  occurred. 

G.  CHANGE  DETECTION 

Previous  change  detection  experiments  involving  virtual  worlds  are  limited 
in  scope  and  size.  When  compared  to  this  research  teams’  military-focused 
objective,  the  environments  used  throughout  the  literature  are  quite  small.  In 
most  cases,  the  VE  is  just  a  room,  house,  or  small  city  block.  While  useful  for 
research  purposes,  it  does  not  provide  the  rich  context  necessary  for  military 
training. 

One  experiment  stood  out  among  the  rest  as  having  some  military 
applicability  for  training.  Karacan,  Cagiltay  and  Tekman  (2010)  studied  change 
detection  around  a  square  block  modeled  in  a  virtual  environment.  Their 
environment  resembled  a  city  park.  This  environment  contained  a  sidewalk 
where  avatars  walked  around  the  square.  On  the  inside  of  the  block  was  a  wall 
that  prohibited  participants  from  looking  at  anywhere  but  the  current  stretch  of 
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sidewalk.  As  a  user  moved  around  the  block,  they  saw  houses,  trees,  and  six 
other  objects  next  to  the  sidewalk.  Those  six  objects  included  a  bench,  street 
lamp,  billboard,  fire  hydrant,  mailbox,  and  a  trashcan.  Unbeknownst  to 
participants,  the  trashcan  was  the  object  that  would  change  in  the  experiment. 
The  changes  that  might  occur  to  the  trashcan  included  appearance  of  a  new 
object  (trashcan),  disappearance,  displacement,  and  replacement  of  the  trashcan 
with  a  new  object  (Karacan  et  al,  2010). 


Figure  4.  Virtual  world  (From  Karacan  et  al.,  2010) 


Early  on,  Karacan  et  al.  admitted  that  humans  are  poor  at  detecting 
changes  in  complex  scenes.  The  phenomenon  of  “change  blindness”  can 
increase  with  the  amount  of  activity  in  a  scene  (Karacan  et  al.,  2010,  1305). 
Karacan  et  al.  allowed  participants  to  become  familiar  with  the  virtual 
environment  in  order  to  determine  if  change  detection  could  improve.  Karacan  et 
al.  relied  on  literature  that  predicts  people  are  more  apt  to  notice  change  in 
familiar  environments.  To  measure  the  detection  of  change,  they  measured  a 
participant’s  eye  gaze  on  specific  objects. 
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The  results  of  Karacan  et  al.’s  experiment  revealed  that  familiarity  with  an 
environment  plays  an  important  role  in  detecting  change.  That  familiarity  was  the 
only  statistically  significant  factor  in  detecting  change.  With  regard  to  the  four 
types  of  changes,  Karacan  et  al.  found  that  the  appearance  of  a  new  object 
attracted  the  attention  of  the  largest  number  of  participants.  Object  deletion  also 
gained  the  attention  of  a  number  of  participants;  however,  neither  of  these  factors 
contributed  significantly  to  detecting  change.  Of  note,  both  change  factors 
produced  a  significant  increase  in  gaze  duration  when  compared  to  the  baseline 
environment  without  changes  (Karacan  et  al.,  2010). 

H.  SIGNAL  DETECTION  THEORY 

This  thesis  sought  to  produce  and  test  a  prototype  trainer  for  improving 
combat  soldiers’  change  detection  skills.  While  eye  tracking  demonstrated  value 
in  the  literature,  the  military  lacked  these  laboratory-type  conditions  at  the  small 
unit  level  for  measuring  performance.  At  the  time  of  this  thesis,  there  existed  no 
reliable,  cost-effective  option  to  distribute  eye-tracking  software  for  all  soldiers 
and  marines  for  use  with  VBS2.  As  a  result,  the  research  team  needed  to 
determine  a  proven  metric  to  assess  performance  at  change  detection  in  VBS2. 
Signal  Detection  Theory  (SDT)  provided  just  the  type  of  objective  measurements 
necessary  to  accomplish  this  mission. 


Classifications  of  Signal  and  Response 


State  of  the  Environment 

Signal 

Noise 

Response 

Yes 

(Present) 

Hit 

False  Alarm 

No  (Absent) 

Miss 

Correct 

Rejection 

Table  1 .  Signal  Detection  Theory  states  and  responses 
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Classic  Signal  Detection  Theory  (SDT)  presents  a  participant  with  two 
types  of  situations.  In  one  case,  there  is  no  signal  present.  The  literature  refers 
to  this  condition  as  “noise  trials.”  In  the  other  case,  there  are  signals  in  the  noise. 
These  are  “signals  plus  noise  trials”  (Proctor  &  Van  Zandt,  2008).  In  SDT,  a 
signal  is  anything  intended  for  detection  within  an  experiment.  Some  examples 
of  potential  signals  include  a  change  in  the  environment,  the  signature  of  an 
explosive  device,  and  human  or  physical  targets.  There  is  no  requirement  that  a 
signal  be  visual.  In  fact,  many  SDT  experiments  measure  sensitivity  to  auditory 
stimulus  (Green  &  Swets,  1966). 

Table  1  outlines  the  two-by-two  matrix  of  possible  SDT  classifications. 
Signal  Detection  Theory  primarily  focuses  on  hits  and  false  alarms.  While 
misses  and  correct  rejections  are  important,  they  are  just  the  complement  of  hits 
and  false  alarms.  An  experiment  normally  assesses  a  participant’s  sensitivity  to 
signals.  The  higher  the  hit  rate  and  lower  the  false  alarm  rate,  the  better  the 
sensitivity  (Proctor  &  Van  Zandt,  2008). 

Signal  Detection  Theory  also  addresses  the  idea  of  bias.  For  instance,  if  a 
person  is  very  intent  on  detecting  a  signal,  they  may  also  be  more  apt  to  commit 
a  higher  number  of  false  alarms.  In  this  case,  we  call  a  participant  “biased” 
towards  responding  that  a  signal  is  present  (Proctor  &  Van  Zandt,  2008,  p.91). 
Bias  can  skew  results,  especially  in  small  sample  sizes  where  one  or  two 
participants  with  bias  greatly  affect  the  overall  results. 

I.  TRAINING  CONSIDERATIONS 

In  a  recent  U.S.  Army  Research  Institute  Report,  Christopher  Vowels 
(2010)  reminds  readers  that  soldiers  must  remain  alert  for  changes  in  their 
environment.  His  paper  focuses  on  threat  detection  and  how  the  Army  must 
create  effective  ways  to  train  for  emergent  threats.  He  identifies  many  of  the 
concerns  of  this  thesis  as  they  relate  to  threat  detection:  vigilance,  attention, 
change  detection,  and  recognition  of  threats,  just  to  name  a  few. 
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Vowels  recommends  Virtual  Battlespace  2  (VBS2)  by  name,  and  suggests 
that  “synthetic  learning  environments,  such  as  simulations  or  online  interactive 
programs,  can  serve  as  the  conduit  to  achieve  any  further  training,  regardless  of 
unit  location”  (Vowels,  2010,  p.7).  He  elaborates  on  the  objectives  of  threat 
detection  training.  Three  of  these  objectives  are  the  reduction  of  errors,  flexibility 
of  the  skills,  and  increased  skill  retention  (Vowels,  2010,  p.6).  These  goals 
became  objectives  for  the  change  detection  trainer  developed  as  part  of  this 
thesis  research. 

Finally,  Vowels  mentions  three  relevant  theories  for  training  threat 
detection:  Active  Control  of  Thought  -  Rational  (ACT-R),  Signal  Detection  Theory 
(SDT),  and  the  Recognition-Primed  Decision  Model  (Vowels,  2010,  pp.  17-22). 
This  thesis  uses  SDT. 

He  explains  that  a  training  participant’s  ability  to  manipulate  visual  space 
and  having  an  understanding  of  their  role  in  that  space  are  critical  to  the  value  of 
the  training  (Vowels,  2010,  pp.  22-23).  The  key  to  improving  the  value  of  threat 
detection  training  is  to  incorporate  the  experiences  of  the  current  force  into 
tangible  tools  to  help  improve  performance  (Vowels,  2010,  p.  23). 

Finally,  the  time  available  for  training  is  a  limited,  finite  resource. 
Deployment  cycles  and  mandatory  pre-deployment  training  objectives  dictate  the 
amount  time  allocated  for  each  task.  Vowels  suggests  that  using  virtual  training 
that  leverages  video  games  is  an  effective  way  to  train  in  garrison  or  while 
deployed  (Vowels,  2010,  p.  23).  He  also  lists  simulations  and  video  games  as 
his  first  recommended  training  format,  explaining  that  there  is  evidence 
supporting  increased  cognitive  skills  from  playing  video  games  (Vowels,  2010, 
p.  31).  Evidence  of  improved  skill  was  demonstrated  by  Orvis  et  al.  (2009,  2010) 
and  Green  and  Bavelier  (2003).  The  opportunity  and  challenge  of  developing  a 
virtual  training  environment  motivates  this  research  team.  Pursing  the  research 
goals  outlined  by  Christopher  Vowels  guides  this  research  effort. 
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III.  METHODOLOGY 


A.  INTRODUCTION 

The  objective  of  this  thesis  was  to  develop  a  change  detection-training 
tool.  In  order  to  meet  this  objective,  the  research  team  looked  at  a  variety  of 
human  performance  factors  and  system  capabilities  needed  to  develop  the 
training  tool.  First,  it  was  necessary  to  identify  the  tasks  that  would  stimulate 
change  detection  training.  After  identifying  key  tasks,  the  team  chose  an 
appropriate  simulation  to  train  those  tasks.  Developing  a  scenario  compatible 
with  the  simulation  was  an  important  next  progression.  Finally,  the  research 
team  evaluated  potential  user  interfaces  and  determined  methods  for  scoring 
trainees  within  the  change  detection  trainer.  Considering  all  these  factors,  the 
team  created  a  change  detection  trainer  and  designed  an  experiment  to  test  the 
apparatus.  The  research  team  created  “training”  and  “no  training”  conditions  for 
the  experiment. 

1.  Tasks 

Designing  a  distributable  training  tool  focused  on  memory  and  change 
detection  begins  with  the  identification  of  training  objectives  and  supporting 
tasks.  Developers  codify  these  objectives  into  requirements.  For  the  purpose  of 
this  thesis,  the  research  team  served  as  the  designers,  developers,  trainers,  and 
evaluators  for  this  tool. 

Initially,  the  team  identified  five  overarching,  real-world  training  tasks  that 
could  benefit  from  an  individual  desktop  virtual  training  environment.  Task 
improvement  was  the  ultimate  objective  of  this  task  trainer.  The  specific  tasks 
were: 

•  Change  detection 

•  Pattern  recognition 
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•  Spatial  memory  encoding  and  recall  or  recognition 

•  Task  memory  encoding,  recall  and  task  performance 

•  Event  memory  encoding  and  recall  or  recognition 

As  presented  in  the  introduction,  the  COE  places  soldiers  and  marines  in 
a  unique  mission  environment.  A  typical  daily  patrol  during  COIN  operations  will 
last  eight  to  twelve  hours.  Within  this  period,  a  soldier  or  marine  may  observe 
hundreds  of  events  and  thousands  of  individuals.  Additionally,  it  is  common  for  a 
unit  to  revisit  the  same  terrain  daily,  sometimes  for  months  at  a  time.  These 
conditions  highlight  the  importance  of  memory,  change  detection,  and  pattern 
recognition.  A  mission  scenario  most  appropriate  for  testing  these  skills  is  a  daily 
patrol  in  an  area  frequently  visited  by  friendly  forces.  The  headquarters  element 
provides  soldiers  with  Priority  Intelligence  Requirements  (PIR)  specific  to  unit 
mission  priorities.  Some  of  these  requirements  include  be  on  the  lookout  (BOLO) 
vehicle  descriptions,  target  names  or  photographs,  possible  enemy  locations, 
possible  IED  locations,  current  enemy  tactics,  techniques,  and  procedures  (TTP), 
previous  significant  activities  (SIGACT)  information,  friendly  unit  positions  and 
missions,  and  many  other  relevant  details  for  the  mission. 

Within  this  basic  scenario,  a  virtual  trainer  can  test  several  top-level  tasks 
that  a  trainee  might  need  to  perform.  The  research  team  identified  eight  primary 
tasks: 

1 .  Recognize  physical  environment  pattern  changes  over  time 

2.  Recognize  social  pattern  changes  in  an  environment  over  time 

3.  Recognize  suspicious  vehicles  from  descriptions  or  photographs 

4.  Recognize  individuals  from  photographs  or  sketches 

5.  Recognize  assigned  boundaries  of  an  operating  environment  (OE) 

6.  Recall  and  narrate  personal  social  interactions  accurately 

7.  Report  recent  high  intensity  activity  events  accurately 

8.  Recognize  answers  to  a  list  of  intelligence  requirements 

The  team  incorporated  Tasks  #1  and  #2  into  the  trainer.  To  a  lesser 
degree,  the  tool  challenges  trainees  on  tasks  #3-5,  but  only  through  recognition 
of  changes.  Tasks  #6-8  are  beyond  the  scope  of  this  experiment;  however,  they 
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are  extremely  useful  skills  that  a  virtual  environment  could  train.  Each  of  these 
top-level  tasks  supports  the  following  examples  of  tactical  tasks: 

1 .  Report  any  changes  in  population  activity  pattern. 

2.  Report  any  changes  in  vehicle  traffic  pattern. 

3.  Clear  intra-sector  routes  of  lEDs,  confirm  or  deny  possible  IED 
locations. 

4.  Provide  status  of  local  law  enforcement. 

5.  Report  on  any  recent  moves  of  families  into  or  out  of  sector 
(resettlement). 

6.  Report  any  new  business  activities  or  new  business  openings, 
particularly  clinics  and  Internet  cafes. 

7.  Immediately  report  any  SIGACTS  accurately,  and  those  which  are 
Commander’s  Critical  Intelligence  Requirements  (CCIR) 

8.  Capture  or  kill  any  High  Value  Targets  (HVT) 

9.  Impound  or  pursue  if  necessary  any  BOLO  vehicles 

10.  Provide  full  patrol  debrief,  including  names,  photographs,  and 
significant  PIR,  particularly  meetings  with  individuals  of  high  influence 

1 1 .  Coordinate  with  adjacent  units  for  any  activities  that  may  affect  other 
sectors. 

12.  Provide  debrief  on  current  tribal  boundaries  and  leadership. 

13.  Provide  debrief  of  current  infrastructure  status  and  any  changes 
Sewage-Water-Electricity-Trash-Health  (SWEAT-H). 

The  research  team  incorporated  the  first  six  (6)  of  these  thirteen  (13) 
tactical  tasks  in  the  change  detection-training  tool.  The  other  seven  (7)  tasks, 
excluded  from  the  current  tool  because  they  were  beyond  the  scope  of  this 
thesis;  however,  virtual  training  provides  a  platform  capable  of  training  these 
skills. 


2.  Simulator:  Choosing  a  Desktop  Virtual  Environment 

After  defining  the  tool’s  objectives  and  the  tasks  that  achieve  those 
objectives,  the  team  needed  to  select  a  virtual  environment.  Three  options 
emerged  as  most  likely  to  meet  the  need:  America’s  Army,  Delta  3D,  and  Virtual 
Battlespace  2  (VBS2) 
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a.  America’s  Army 

Originally  developed  at  the  Naval  Postgraduate  School  (NPS), 
America’s  Army  presented  a  viable  medium  to  develop  a  desktop  virtual  training 
environment.  America’s  Army  is  an  open  source  model  for  development  within 
the  United  States  Army.  It  presents  trainees  with  uniforms  and  vehicles  familiar 
to  them.  The  downside  of  developing  a  tool  in  America’s  Army  is  the  lack  of 
technical  support  and  the  fact  that  very  few  people  in  the  Army  are  using  this 
software  for  training.  America’s  Army  is  primarily  a  recruiting  tool. 

b.  Delta  3D 

Next,  the  research  team  evaluated  Delta  3D  as  a  potential 
candidate  for  a  development  environment.  The  Modeling,  Virtual  Environments, 
and  Simulations  (MOVES)  Institute  at  NPS  developed  Delta  3D  as  an  open 
source  gaming  engine.  MOVES  staffed  a  small  Delta  3D  development  team, 
which  assists  students  with  projects  and  thesis  work. 

The  research  team  developed  the  first  prototype  trainer  using  Delta 
3D.  Delta  3D  provided  the  flexibility  necessary  to  implement  a  training  tool  from 
scratch.  There  were  no  limitations  due  to  proprietary  software,  and  the 
environment  had  the  added  advantage  of  being  programmable  in  Python  or  C++, 
two  languages  familiar  to  the  researchers.  This  combination  of  flexibility, 
availability,  and  usability  made  the  tool  an  attractive  option. 

During  development  of  the  prototype,  the  team  used  Blender  ®  to 
create  many  models.  While  Blender  ®  was  very  useful  for  the  creation  of  exact 
changes  it  was  also  labor  intensive.  Given  the  limited  time  to  conduct  this  thesis 
work,  it  had  the  potential  to  extend  development  timelines  beyond  acceptable 
limits. 

Ultimately,  the  downside  to  Delta  3D  was  the  fact  that  very  few 
people  in  the  Army  are  using  that  gaming  engine  for  training  applications.  The 
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image  library  for  the  engine  contained  pictures  of  Marines,  but  no  agents  for 
Army  soldiers.  This  oversight  was  not  a  trivial  fact  when  contemplating  the 
prospects  of  justifying  a  training  apparatus  in  front  of  Army  flag  officers.  Finally, 
the  Army  recently  invested  large  sums  of  money  in  Virtual  Battlespace  2™,  and 
the  research  team  felt  the  momentum  of  that  simulation. 

c.  Virtual  Battlespace  2™ 

VBS2  does  not  share  the  advantages  of  an  open  game  engine 
such  as  Delta3D.  However,  the  research  team  determined  that  its  flexible 
scenario  editor,  significant  number  of  pre-built  model  assets,  and  pre-built  terrain 
would  offer  the  best  capabilities  for  design  of  the  virtual  testing  environment. 
Although  the  proprietary  source  code  was  inaccessible,  VBS2  did  offer  the  ability 
to  run  customized  scripts  that  could  be  bound  to  keyboard  commands. 
Additionally,  this  simulation’s  3D  graphics  fidelity  sufficed  to  model  a  plausible, 
current  operational  setting. 

The  research  team  tested  VBS2  during  a  class  project  for  OA3302, 
Research  Methods  for  Performance  Assessment,  at  NPS.  The  experiment 
involved  target  identification  within  the  VBS2  virtual  environment.  Developing 
that  experiment  taught  the  research  team  the  basics  of  VBS2  scripting.  The 
experience  enabled  the  implementation  of  additional  capabilities  beyond  the  “out- 
of-the-box”  functionality  of  VBS2.  Specifically,  VBS2  offered  several  methods  to 
test  the  intersection  of  a  user’s  view  vector  with  a  flagged  object.  By  testing 
several  simple  scripts  employing  these  methods,  the  team  developed  an 
accurate  and  dependable  algorithm  for  users  to  identify  a  target  in  their  view  with 
a  key-press.  Additionally,  the  team  discovered  how  to  extract  the  appropriate 
data  from  the  simulation.  VBS2  scripts  copied  the  data  to  text  files  for  analysis. 
Ultimately,  the  research  team  found  that  VBS2  scripting  offered  enough 
capability  for  the  final  virtual  testing  environment. 
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The  extensive  testing  done  by  the  research  team  proved  that  VBS2 
was  viable  tool  for  change  detection  training.  The  fact  that  the  Army  widely  used 
VBS2  for  a  multitude  of  training  tasks  solidified  its  selection.  Moving  forward,  all 
work  utilized  VBS2,  and  with  this  decision  out  of  the  way,  the  research  team 
shifted  focus  to  scenario  development  and  experimental  design. 

3.  Scenario 

a.  VBS2  Scenarios 

The  team  created  a  “baseline”  and  two  (2)  “change”  scenarios  for 
the  experiment.  The  roads  and  infrastructure  in  the  environment  were  the  same 
regardless  of  scenario.  The  scenario  guided  the  participants  along  the  same 
route  through  the  virtual  environment  every  session. 


Figure  5.  Routes  through  the  virtual  environment 


The  team  determined  that  a  participant  would  occupy  the  gunner’s  turret  in 
a  High  Mobility  Multi-Wheeled  Vehicle  (HMMWV).  The  artificial  intelligence  (Al) 
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piloted  the  simulation  along  assigned  waypoints.  The  developers  placed  these 
waypoints  in  the  virtual  environment  using  the  VBS2  scenario-editing  tools.  By 
controlling  route  navigation,  the  researchers  eliminated  a  source  of  variance  that 
would  have  undoubtedly  arisen  if  participants  had  been  allowed  to  self-navigate 
the  environment.  The  research  team  specified  a  speed  of  10  km/h  for  the  vehicle 
Al  in  order  to  allow  users  a  sufficient  amount  of  time  to  observe  the  environment 
and  identify  changes.  The  gunner  position  of  the  HMMWV  was  an  ideal 
observation  point  because  it  is  at  the  center  of  the  vehicle  and  provides  a  360- 
degree  freedom  of  movement  (FOM).  A  participant’s  field  of  view  (FOV)  in  the 
simulation  was  60-degrees  horizontally  and  38-degrees  vertically,  a  16:10  aspect 
ratio. 

A  plausible  operational  environment  necessitated  the  selection  of 
pre-existing  terrain  within  VBS2.  The  research  team  chose  VBS2’s  “As 
Samawah  (GAA)”  terrain  because  of  its  similarity  to  a  typical  Iraqi  urban 
environment.  Although  several  environments  depict  Afghanistan  in  VBS2,  the 
research  team  determined  that  the  scarcity  of  objects  in  these  particular  terrains 
did  not  provide  a  sufficient  amount  of  “noise”  for  the  scenarios. 

A  pilot  study  conducted  as  part  of  coursework  for  Research 
Methods  for  Performance  Assessment  showed  that  a  target  every  20-30 
seconds  resulted  in  sustained,  high-level  user  performance.  A  pilot  study 
conducted  in  the  Human  Factors  of  Systems  Design  course  revealed  a  vigilance 
decrement  by  participants  occurring  between  12-25  minutes.  Accounting  for 
these  two  results,  the  team  limited  scenarios  for  this  experiment  to  12  minutes, 
excluding  any  instruction.  The  team  set  the  number  of  changes  in  a  “signal  plus 
noise”  scenario  to  thirty  (30). 
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b.  Designing  for  Vigilance 


In  her  research  on  vigilance,  arousal,  and  habituation,  Jane  F. 
Mackworth  described  several  features  to  reduce  habituation  during  user  task 
performance  (Mackworth,  1968,  pp.  309-316): 

•  Background  events. 

•  Presentation  of  other  stimulus. 

•  Rest  pauses. 

•  Knowledge  of  Results  (KR) 

In  order  to  maintain  user  vigilance,  scenario  design  included 
several  of  these  features.  Scenarios  contained  three  stopping-points  for  the 
HMMWV  requiring  the  participant  to  enter  the  short  keyboard  commands  “1”  and 
“9”  to  continue.  Designers  added  triggered  sound  events  throughout  the 
scenarios,  activated  when  the  trainee’s  vehicle  entered  pre-defined  areas  in  the 
VE.  Some  triggers  created  ambient  noise  events  such  as  “Call  to  Prayer”  sounds 
or  traffic  noises,  while  other  triggers  initiated  louder  noises,  such  as  a  man  yelling 
on  a  loudspeaker  or  a  truck  horn.  The  team  scripted  ambient  traffic  noise 
including  both  other  road  vehicles  and  helicopters  flying  over  the  route.  The 
design  did  not  space  targets  evenly  throughout  the  route.  Instead,  the  design 
implemented  fluctuation  in  both  actual  targets  and  level  of  object  noise 
throughout  all  scenarios.  Additionally,  the  team  implemented  KR  with  a  “ding” 
sound  that  would  play  if  a  participant  correctly  identified  a  change. 

c.  Landmarks  and  Spatial  Navigation 

Since  change  detection  is  notoriously  difficult  (Rensink,  2002),  the 
researchers  implemented  various  recommended  techniques  to  enhance  spatial 
knowledge  acquisition.  When  designing  virtual  environments,  Darken  and  Sibert 
suggest  the  use  of  real  world  design  techniques  (Darken  &  Sibert,  1996).  These 
provide  an  environment  that  facilitates  learning  spatial  knowledge  related  tasks. 
Although  Darken  and  Sibert  specifically  address  navigation,  the  design  team 
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extended  these  principles  to  scenario  design.  Facilitating  efficient  acquisition  of 
spatial  knowledge  was  the  design’s  ultimate  goal.  Darken  and  Sibert  suggest 
organizational  principles  that  should  be  applied,  including  dividing  the  world  into 
“distinct,  small  parts,”  organizing  these  parts  using  a  “simple  organizational 
principle”  such  as  a  grid,  and  providing  frequent  directional  cues  (Darken  & 
Sibert,  1996,  p.3).  In  order  to  maintain  spatial  awareness  throughout  virtual 
environments,  they  encouraged  the  use  of  maps  as  a  fundamental  tool.  The 
researchers  included  a  paper  overhead  map  of  the  scenario  area  with  labeled 
landmarks  at  the  participant’s  station  (Figure  5).  Additionally,  the  researchers 
chose  a  scenario  area  that  could  be  broken  into  four  distinct  parts.  Improving  a 
participant’s  spatial  memory  performance  guided  the  selection  of  this  particular 
piece  of  terrain. 

Steck  and  Mallot  advocated  the  implementation  of  global  and  local 
landmarks  in  the  virtual  environment  in  order  to  assist  navigation  in  the  virtual 
world  (Steck  &  Mallot,  2000).  Their  experiment  showed  that  participants 
encoded  landmark  information  into  memory,  showing  some  level  of  improved 
spatial  memory  acquisition.  Implementing  landmarks  in  this  scenario  had  the 
potential  to  enhance  spatial  awareness,  so  the  research  team  chose  to  include  a 
few  unique  landmark  locations.  These  landmarks  included  a  bridge,  a  river,  and 
a  Iraqi  Police  recruiting  area. 

d.  Changes  in  the  Scenarios 

Next,  the  research  team  determined  what  changes  were 
appropriate  for  this  study.  VBS2  contained  hundreds  of  objects  for  insertion  into 
scenario  design.  To  maintain  operational  plausibility,  the  team  chose  objects  to 
replicate  changes  that  made  sense  in  the  context  of  a  busy  Middle  Eastern  urban 
environment.  Both  authors  have  a  total  of  36  months  in  Iraq  and  Afghanistan. 
This  combat  experience  informed  the  selection  of  changes  for  implementation  in 
each  scenario. 
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As  change  detection  is  a  difficult  task  in  any  circumstance,  the  team 
decided  on  conspicuous  changes  that  would  not  be  overly  difficult  to  see  in  the 
virtual  environment.  The  team  did  not  attempt  to  hide  objects.  The  design  team 
placed  the  changed  objects  a  sufficient  distance  apart  from  each  other  to  prevent 
unintentional  detections. 

An  expert  team  of  soldiers  with  combat  patrol  experience  in  Iraq 
rated  each  change  for  a  threat  value  on  an  ordinal  scale  of  one  to  five.  The 
researchers  presented  the  experts  the  before  and  after  screenshots  for  each 
change  and  asked  the  experts  to  assign  a  threat  value  using  the  following 
guidelines: 

•  Rating:  5  -  Immediate  action  required  to  react  to  high  threat; 
followed  by  report  to  higher 

•  Rating:  4  -  Significant  threat,  report  to  higher  and  then  investigate 
further  with  caution 

•  Rating:  3  -  Possible  threat  or  significant  activity,  report  to  higher 
when  convenient 

•  Rating:  2  -  Little  or  no  threat  but  significant  enough  activity  to 
include  in  a  patrol  debrief  following  the  mission 

•  Rating:  1  -  No  threat. 

The  research  team  averaged  all  expert  ratings  and  determined  the 
standard  deviations.  Threat  rating  determined  the  rank  of  each  change  in  the 
scenario.  The  team  blocked  the  lowest  ten  (10)  ratings  as  “low  threat,”  the 
highest  ten  (10)  ratings  as  “high  threat,”  and  the  remaining  ten  (10)  ratings  as 
“medium  threat.”  Developers  assigned  variable  names  to  changes  prior  to 
subject  matter  expert  (SME)  evaluations.  Variable  names  had  no  correlation  to 
SME  evaluated  threat  levels. 

Appendix  K  has  a  complete  list  of  before  and  after  screen  shots  of 
the  changes  used  in  each  scenario.  Appendix  I  contains  the  expert  rating  of 
each  change.  Again,  the  research  team  warns  against  interpreting  any  threat 
level  based  on  variable  names. 
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Figure  6.  Example  of  baseline  environment  before  change 


Figure  7.  Example  of  changed  environment  (same  locations  as  Figure  6) 
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# 

Variable  Name 

Before 

After 

Threat 

Rating 

1 

tire_1 

Two  men 

Stack  of  tires 

High 

2 

burnt  truck  1 

SUV  in  good  condition 

Burnt  SUV 

Medium 

3 

propane  tank  sales  1 

Empty  garage 

Man  and  boy  with  propane  tanks  in  garage 

Low 

4 

broken_wall_1 

Solid  stone  wall 

Broken  wall 

Low 

5 

dirt _pile_l 

Empty  lot 

Bulldozer  and  dirt  pile 

Low 

6 

observer_l 

Empty  balcony 

Man  on  balcony 

Medium 

7 

vid_camera_l 

Man  with  motorcycle  shop 

Motorcycles  gone,  same  man  with  video 

camera 

High 

8 

running_man_l 

Group  of  men 

Man  runs  away  from  HMMWV 

High 

9 

ied_box_l 

Carpet  salesman  and  a  van 

Salesmen,  carpet  and  van  gone.  New  box 
present 

Medium 

10 

ied_supply_man_l 

Man  selling  teddy  bears  (toys)  to  children 

Children  gone,  man  selling  batteries,  cell 
phones,  and  fuel  tanks. 

Low 

ii 

odd_drums_l 

Nothing 

Four  (4)  colored  oil  drums 

High 

12 

vbied_van_l 

Man  selling  plants  out  of  a  blue  van 

No  people  present.  Lowered  while  van, 
possibly  weighted  by  explosives. 

Medium 

13 

ied_patch_l 

Nothing 

Small  two-foot  diameter  patch,  possibly  an 
implanted  IED. 

Medium 

14  burkha_girl_l 

Three  females  in  western  dress 

Three  females  in  full,  black  burkha 

Low 

15 

ruined_house_l 

House  with  SUV  and  three  females  in  western 

dress 

House  demolished,  several  suspicious 
individuals  in  the  background. 

Medium 

16 

broken_wall_2 

Iraqi  Police  (IP)  with  IP  truck  next  to  a  barrier 

IP  gone,  one  barrier  destroyed,  white  sacks - 
possible  emplaced  explosive 

Low 

17 

bad_trash_pile_l 

Several  workers  and  heavy  machinery 

Workers  and  machinery  gone,  new  rubble  with 
suspicious  barrel  extending  into  the  road 

Medium 

18 

broken_wa  1  l_ied_l 

Solid  wall 

Wall  broken,  dark  patch  indicating  possible  IED 
at  the  site  of  the  break 

Medium 

19 

broken_guardrail_l 

Solid  unmolested  guardrail 

Broken  guardrail  in  two  separate  sections 

Medium 

20 

bad_curb_l 

Nothing 

Low  curb  with  a  broken  section  indicating 
possible  IED  emplacement. 

High 

21 

reporter_l 

Group  of  me  picnicing  at  a  roadside  cafe 

Reporter  and  cameraman  interviewing  men  at 
the  roadside  cafe 

Medium 

22 

bad_bag_pile 

Two  (2)  men  and  four  (4)  black  bags 

No  men  and  twelve  (12)  black  bags  stacked 
tightly 

High 

23 

bad_barrels_l 

White  truck  with  three  barrels,  group  of  six  (6) 
men  wandering 

Six  (6)  barrels,  no  truck,  group  of  men  gone, 
one  (1)  man  observing  the  barrels  from  a 
concealed  position 

High 

24 

bad_rubble_l 

Large  propane  tank,  no  rubble  present 

Large  propane  tank  burried  in  a  new  rubble  pile 

High 

25 

broken_fence_no_i  p_l 

IP  HMMWV  behind  barbed-wire  fence 

HMMWV  gone  and  half  of  fence  torn  down 

Low 

26 

burried_tank_l 

Large  septic  tank  on  the  left  side  of  the  road 
across  from  IP  recruiting  station 

Septic  tank  burried  on  the  right  side  of  the  road 
next  to  the  IP  recruiting  station 

High 

27 

new_conta  i  ners_l 

Large  white  truck  moving  in  an  empty  supply 
yard 

Truck  is  gone,  six  (6)  new  large  shipping 
containers  in  the  supply  yard 

Low 

28 

poster_wall_l 

Unmolested  T-Wall  barriers 

Two  (2)  large  propaganda  posters  on  wall 

Low 

29 

concrete_ied_l 

Concrete  wall 

Broken  concrete  wall,  large  concrete  block 
present 

High 

30 

bad_mixer_l 

Nothing 

Large  concrete  mixer  on  the  side  of  the  road, 
parked  in  an  unusual  location 

Low 

Table  2.  Scenario  #1  change  list 
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# 

Variable  Name 

Before 

After 

Threat 

Rating 

1 

truck_wreck 

Small  pick-up  truck 

Large  truck  wreck 

Medium 

2 

fridge_no_play 

Children  playing 

Children  gone,  broken  refridgerator,  and 
suspicious  objects  present 

High 

3 

suicide_bomber 

Group  of  five  (5)  men  between  appartment 
buildings 

Men  gone,  man  with  suspicious  vest  is  visible  in 
the  window  of  one  appartment\ 

High 

4 

dead_body 

Two  (2)  IP's  patrolling  on  foot,  three  (3)  civilians 
walking 

IP  gone,  civilians  looking  at  dead  body  on  the 
ground 

High 

5 

bomb_maker 

Motorcycle  outside  the  garage  with  a  man 

Man  and  motorcycle  in  garrage  with  a  large 
amount  of  fuel  cans 

Low 

6 

new_rock 

Nothing 

Suspicious,  large  rock 

High 

7 

new_van 

Empty  garage  front  near  IP  checkpoint 

Blue  van  parked  near  IP  checkpoint  in  front  of 
garages 

Low 

8 

boarded_up_shop 

Man  with  computers  in  garage 

Garage  closed  up  with  sheet  metal  with  dirt 
tracks  leading  out 

Medium 

9 

weird_dudes 

Man  moving  crates  and  a  utility  truck 

Crates,  man  and  truck  gone.  Four  (4)  suspicious 
men  observing  the  road. 

Low 

10 

press_man 

Group  of  five  (5)  men  wandering  around 

Men  gone,  man  with  press  jacket  and  while 
press  SUV  present 

Low 

11 

bomb_car 

Group  of  women  in  burkha,  empty  street 

Women  gone,  abandoned  white  vehicle  in  the 

street 

High 

12 

ba  c  k_to_b  a  c  k_va  ns 

A  man  standing  by  a  garage  with  two  red 
vehicles 

Red  vehicles  gone,  vans  backed  up  to  each 
other  to  conceal  man’s  activity  in  the  garage 

Medium 

13 

UN_sacks 

Empty  garage 

Garage  full  of  UN  food  sacks 

Medium 

14 

ice_cream_dude 

Man  selling  ice  cream  to  children  from  cart 

Children  gone,  cart  gone,  man  hiding  in 
doorway 

Low 

15 

bad_pile 

Busy  cafe  front  with  numerous  people 

All  people  gone,  large  bags  in  front  of  cafe 

High 

16 

new_b  oxes_n  o_p  h  one_b  ooth 

Phonebooth 

Phonebooth  gone,  boxes  stacked  by  roadside 

Medium 

17 

new_tractor 

White  car 

White  car  gone,  tracktor  in  its  place 

Low 

18 

nobody_ia_cp 

IA  checkpoint  with  one  (1)  HMMWV  and  two  (2) 
personnel  cheking  trucks 

Checkpoint,  HMMWV,  and  IP  gone,  trucks 
moving  freely 

Low 

19 

ne  w_c  a  r_o  pe  n_str  eet 

Road  blocked  by  barriers 

Barriers  removed,  green  car  using  opened 

street 

Medium 

20 

broken_wall 

T-Wall  blocking  a  side  road 

One  barrier  broken,  suspicious  patch  on  the 
ground 

Medium 

21 

turned_truck 

Rusted  hulk  of  truck  sitting  upright  on  wheels 

Rusted  truck  turned  on  its  side  and  positioned 
next  to  the  road 

Medium 

22 

jingle_truck 

Empty  road 

Jingle  truck  present  on  the  road 

Low 

23 

new  rubble  no  k i ds 

Kids  playing  in  courtyard 

Kids  gone,  courtyard  full  of  metal  scraps 

Medium 

24  wall_truck 

Empty  side  road  behind  T-Wall  barrier 

Truck  parked  directly  behind  T-Wall  barrier 

High 

25 

ip_suicide_bomber 

Empty  alley  across  from  IP  recruiting  station 

Man  with  strange  vest  across  from  IP  recruiting 

station 

High 

26 

propane_tanks 

Man  moving  shipping  boxes  into  large  truck 

Man  and  truck  gone,  propane  tanks  placed  near 
road 

Medium 

27 

bad_cart 

Donkey  and  cart  with  a  man 

Donkey  and  man  gone,  cart  full  of  suspicious 
objects 

High 

28 

new_cars 

Empty  supply  yard 

Four  (4)  new  cars  in  the  supply  yard 

Low 

29 

van_bomb 

Yellow  van  parked  near  the  road 

Weighted  yellow  van  now  in  a  garage  near  the 
road 

Low 

30 

new_barrels 

Empty  street  corner 

Three  (3)  new  purple  barrels 

High  J 

Table  3.  Scenario  #2  change  list 
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e.  In-Simulation  Instruction 

All  scenarios  began  with  in-simulation  voiced  instruction.  The 
instruction  was  more  significant  in  the  first  week  and  minimal  by  the  second  week 
(Appendix  G).  The  early  scenarios  emphasized  the  parameters  of  the  change 
detection  algorithm  in  detail  and  with  a  short  test  demonstration  of  the  15-degree 
left  and  right  azimuth  and  50-meter  distance  limits.  Each  scenario  ended  with 
voiced  instruction  on  expectations  for  the  participant’s  next  session.  The  final, 
twelfth,  scenario  concluded  with  an  in-simulation  survey  of  three  questions. 

f.  In-Simulation  Automated  Tutorial:  the  Effect  of 
“Training”  vs.  “No  Training” 

Richards  et  al.  stated  (2002,  p.  223),  “Human  orientation  requires 
one  to  remember  spatial  arrangements  of  visual  landmarks  and  visualize  them 
from  different  perspectives.”  In  their  experiment,  they  concluded  that  subjects 
eventually  develop  effective  strategies  for  remembering  the  spatial  arrangement 
of  objects.  They  also  found  that  teaching  these  strategies  in  advance  greatly 
improved  performance  at  remembering  the  spatial  arrangement  of  objects. 
Additionally,  these  strategies  worked  as  an  equalizer  for  those  less  apt  to  quickly 
adapt  to  the  virtual  environment.  Just  as  Orvis  et  al.  (2010)  concluded, 
pretraining  provided  less  experienced  users  with  a  method  to  improve 
performance. 
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Figure  8.  Screenshot  of  in-simulation  tutorial 


For  this  thesis,  the  “training”  group  received  short  instruction  within 
the  same  virtual  environment.  The  team  created  the  tutorial  on  a  different  part  of 
the  As  Samawah  terrain.  Prior  to  the  first  session  and  the  seventh  session 
(Week  1  and  Week  3)  the  “training”  group  experienced  the  short  tutorial.  The  “no 
training”  group  received  no  tutorial  instruction.  The  instructional  scenario  in  week 
1  was  different  from  the  instructional  scenario  in  week  3.  The  week  1  training 
focused  on  basic  change  detection  strategies  in  the  virtual  environment,  while  the 
session  in  week  3  taught  some  more  advanced  techniques  for  success.  The 
research  team  designed  each  training  session  just  like  the  experimental 
scenarios;  however,  the  team  used  different  terrain  and  different  changes  from 
the  “signal  plus  noise”  scenarios.  Each  instructional  scenario  lasted 
approximately  six  (6)  minutes.  Participants  in  this  group  observed  a  example 
baseline  scenario  with  voiceover  instruction.  Then,  they  immediately  viewed  a 
changed,  signal  plus  noise,  environment  with  additional  voiceover  instruction 
(Appendix  G,  Subsection  B). 
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4. 


User  interface 


The  team’s  objective  for  the  interface  was  one  that  did  not  detract  from  the 
learning  experience.  Therefore,  the  team  chose  a  minimalist  approach  for  input 
requirements.  There  were  four  primary  interface  requirements: 

•  A  visual  presentation  of  the  environment  (output) 

•  A  auditory  presentation  of  the  environment  (output) 

•  A  method  for  participants  to  manipulate  their  view  (input) 

•  A  method  for  participants  to  indicate  they  detected  a  change  (input) 

Due  to  the  desire  for  the  trainer  to  be  distributable,  the  team  chose  to  use 
a  standard  monitor  and  headphones  as  output  devices  to  participants.  These 
devices  were  simple,  cheap,  and  dominated  usability  considerations  when 
compared  to  head-mounted  displays  or  surround  sound. 

For  view  control,  the  design  team  tested  several  options.  As  part  of 
OA3401,  Human  Factors  in  System  Design,  course  requirements,  the  research 
team  tested  the  effectiveness  of  two  input  devices.  These  devices  were  the 
common  personal  computer  (PC)  mouse  and  a  head-mounted  control  device 
called  the  TrackIR  by  Naturalpoint.  The  TrackIR  allowed  six  degrees  of  freedom 
(6DOF)  head  movement  to  manipulate  a  participant’s  view  of  the  virtual  world. 
The  results  of  the  experiment  indicated  that  the  mouse  presented  less  of  a 
learning  curve  for  users.  Even  though  the  TrackIR  was  more  natural  and  closely 
modeled  the  kind  of  head  movements  required  to  observe  real  environments,  the 
TrackIR’s  learning  curve  was  steep.  It  also  had  the  potential  to  detract  from  the 
task  of  learning  to  detect  change.  The  design  team  chose  to  use  a  large, 
wireless  mouse  with  a  special  low  friction  mouse  pad  in  order  to  provide  a 
smooth  mouse  surface  that  would  not  distract  the  user  while  controlling  the 
viewpoint  in  the  simulation. 
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Figure  9.  Experimental  interfaces 


Finally,  the  research  team  explored  various  methods  for  getting  input  on 
confidence  and  detections  from  participants.  During  the  same  experiment  in  the 
Human  Factors  in  Systems  Design  course,  the  research  team  required  a  user  to 
verbalize  their  detections.  This  approach  was  effective,  but  it  placed  a  heavy 
burden  on  both  the  participant  and  observer  to  verify  detections.  An  automated 
solution  was  necessary. 

During  the  experiment  conducted  as  part  of  Research  Methods  for 
Performance  Assessment,  the  team  scripted  a  method  in  VBS2  that  linked 
keyboard  inputs  to  user  decisions.  Text  file  outputs  for  analysis  validated  the 
input  method,  and  the  team  decided  that  using  the  keyboard  for  input  was  best. 
Additionally,  keyboards  supported  the  central  tenet  of  being  distributable  to  the 
entire  Army. 
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5. 


Scoring 


The  team  bound  the  “V,”  “B”  and  “N”  keys  to  high,  medium  and  low 
confidence  selections.  VBS2  scripts  enabled  the  key  binding,  and  the  team  re¬ 
labeled  the  keys  “H,”  “M”  and  “L”  as  representations  for  high,  medium,  or  low 
confidence  levels,  respectively.  The  research  team  instructed  the  participants  to 
press  the  key  corresponding  with  the  participant’s  confidence  that  a  change  was 
in  their  field  of  view.  A  key  press  executed  the  appropriate  script,  registered  a 
button  press,  and  then  checked  for  a  listed  change  using  the  VBS2  isLookingAt 
command  script.  The  method  received  the  participant’s  current  viewpoint  as 
input,  and  the  algorithm  compared  any  objects  in  the  participant’s  field  of  view 
with  an  array  of  known  changes.  The  array  contained  variable  names  for  all 
objects  designated  as  changes  in  each  scenario.  VBS2  scored  a  correct 
detection  as  a  “hit,”  if  an  object  in  the  array  was  within  a  15-degree  arc  and 
closer  than  50  meters.  These  scripts  scored  each  key  press  a  “hit”  or  “false 
alarm”  at  that  confidence  level  in  the  text  file  output.  VBS2  scripts  logged  any 
key  press  by  a  participant,  whether  or  not  a  change  was  present. 

B.  PARTICIPANTS 

1.  Recruiting 

Research  participants  were  recruited  from  the  student  body  at  the  Naval 
Postgraduate  School  (NPS).  The  research  team  recruited  participants  on  a 
voluntary  basis  using  an  Institutional  Review  Board  (IRB)  approved  invitation  e- 
mail.  Screening  criteria  for  participants  included  three  factors.  Because  testing 
scenarios  depicted  a  neighborhood  similar  to  ones  in  Iraq  or  Afghanistan, 
participants  could  have  no  history  of  Post-Traumatic  Stress  (PTS).  Due  to  a  first- 
person  perspective  in  VBS2,  participants  could  have  no  history  of  simulator 
sickness  or  motion  sickness.  In  order  to  eliminate  possible  confounds  due  to 
vision  differences,  participants  needed  to  have  vision  correctable  to  20/20. 
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Sixteen  volunteers  passed  all  screening  criteria.  One  volunteer  began  the 
experiment,  but  did  not  complete  the  study  for  personal  reasons.  The  fifteen 
participants  completing  the  study  came  from  all  branches  of  service  and  included 
two  international  students.  All  fifteen  volunteers  were  males  between  the  ages  of 
28  and  41  years-old.  The  average  age  was  34.  The  team  made  no  conscious 
decision  to  exclude  female  participants.  In  fact,  females  participated  in  pilot 
studies  conducted  as  part  of  coursework  at  NPS  and  did  quite  well  at  change 
detection  tasks.  However,  no  females  volunteered  for  this  experiment. 


Catgory 

Specific  Category 

Total 

Training 

No  Training 

Navy 

6 

3 

3 

Army 

3 

2 

1 

Branch  of  Service 

Marines 

2 

0 

2 

Air  Force 

2 

1 

1 

International 

2 

1 

1 

04 

8 

4 

4 

Rank 

03 

6 

3 

3 

02 

1 

0 

1 

Play  Video  Games 

Yes 

9 

4 

5 

No 

6 

3 

3 

Gamers 

Yes 

7 

2 

5 

No 

8 

5 

3 

Yes 

3 

2 

1 

Ground  Combat  Experience 

No 

12 

5 

7 

Table  4.  Participant  Demographic  Data 


2.  Blocking 

The  research  team  randomly  assigned  participants  between  two 
experimental  groups.  A  control  group  received  no  training  in  the  virtual 
environment.  The  experimental  group  received  a  brief,  three-minute  (3)  change 
detection  tutorial  every  other  week.  The  research  team  blocked  participants  by 
the  categories  listed  in  Table  2.  The  characteristics  most  important  were  branch- 
of-service,  rank,  and  video  game  experience.  The  research  team  determined 
that  video  game  experience  was  more  important  than  whether  or  not  a  participant 
considered  himself  a  gamer.  The  term  gamer  was  largely  subjective  while 
playing  video  games  was  a  nominal  yes  or  no.  Many  participants  commented 
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that  they  did  not  know  what  constituted  being  a  “gamer.”  They  were  not  all 
satisfied  with  Orvis’  definition  mentioned  previously  in  this  report. 

For  the  remainder  of  this  thesis,  the  terms  “no  training”  and  “training” 
describe  the  two  experimental  groups.  These  titles  are  easy  to  remember  and 
adequately  depict  the  difference  between  the  two. 

C.  PROCEDURES 

1.  Before  the  Experiment 

Before  beginning  the  experiment,  each  participant  received  an  e-mail  with 
instructions  based  on  their  experimental  group  and  a  map  of  the  virtual 
environment  (Appendix  A  contains  copies  of  these  materials).  The  research 
team  scheduled  the  first  three  experimental  sessions  for  week  1  prior  to 
beginning  any  experimentation.  This  scheduling  technique  ensured  that 
participants  completed  all  three  sessions  with  approximately  24-hours  between 
sessions,  on  average  (Appendix  H  contains  the  exact  times  between  each 
participant’s  training  sessions). 

Prior  to  beginning  Session#1  of  the  experiment,  participants  read  and 
signed  the  IRB-approved  Informed  Consent  Form  (Appendix  D).  They  also 
completed  a  brief  demographic  survey  (Appendix  E)  used  to  assign  participants 
to  groups  and  determine  some  key  characteristics  for  analysis  of  the  results. 
One  of  the  survey  questions  asked  participants  if  they  thought  that  computer- 
based  simulation  training  was  an  effective  training  tool  for  tactical  training. 
Figure  10  outlines  the  results  of  this  survey. 
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t  Computer  Based  Training 
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Figure  10.  Participant  opinions  of  computer  simulation  based  training 

The  computer  based  simulation-training  question  identified  any  pre¬ 
existing  biases  participants  might  have  about  the  training  they  received  in  VBS2. 
An  overwhelming  majority  agreed,  in  varying  degrees,  that  computer-based 
simulation  training  was  an  effective  method. 

2.  During  the  Experiment 

The  TRADOC  Analysis  Center  -  Monterey  (TRAC-MTRY)  housed  the 
experimental  cubicle  in  their  combat  models  lab.  When  a  participant  arrived  for 
one  of  their  twelve,  15-minute  sessions,  the  research  team  escorted  them  into 
the  testing  cubicle.  The  team  allowed  participants  a  moment  to  get  comfortable 
in  the  chair,  don  the  headphones,  position  their  hands  on  the  keyboard,  and 
grasp  the  mouse.  The  simulation  began  when  a  participant  indicated  they  were 
ready.  Each  session  began  with  in-simulation  instructions  specific  to  that  day 
(Appendix  G). 

Each  week  followed  the  same  sequence  of  three  sessions.  The  first 
session  exposed  a  participant  to  the  “baseline”  environment.  The  research  team 
explicitly  told  users  to  examine  the  environment  so  that  they  could  properly 
identify  changes  in  future  scenarios.  The  second  session  was  another  exposure 
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to  the  same  “baseline”  scenario;  but  the  research  team  concealed  the  fact  that  it 
was  the  “baseline”  scenario.  The  team  instructed  participants  to  look  for 
changes.  Finally,  the  third  and  final  session  of  each  week  presented  one  of  the 
two  “change”  scenarios. 


WEEK 

DAY  1 

DAY  2 

DAY  3 

1 

Baseline,  explicit 
Tutorial  for  Training’ 
group 

Baseline,  concealed 

Change  1 

2 

Baseline,  explicit 

Baseline,  concealed 

Change  2 

3 

Baseline,  explicit 
Tutorial  for  Training’ 
group 

Baseline,  concealed 

Change  1 

4 

Baseline,  explicit 

Baseline,  concealed 

Change  2 

Table  5.  Treatment  Plan  for  Participants 


During  a  given  scenario,  each  participant  scanned  the  road,  both  left  and 
right  to  learn  the  environment  or  detect  changes  in  the  environment.  If  a 
participant  thought  they  discovered  a  change,  they  moved  the  mouse  cursor 
within  15  degrees  and  50  meters  of  the  change  and  pressed  a  confidence  key  on 
the  keyboard.  If  they  were  correct,  they  received  an  audible  tone,  knowledge  of 
results  (KR).  If  they  were  incorrect,  they  received  no  KR. 

The  simulation  consisted  of  four  right-hand  turns.  The  participant  pressed 
the  “1”  and  then  the  “9”  key  to  continue  on  the  waypoint.  Pressing  waypoint 
buttons  was  a  secondary  task  to  keep  the  user  engaged  in  the  scenario  and  keep 
the  simulation  on  track.  The  simulation  ended  when  the  HMMWV  reached  the 
end  of  the  fourth  leg  of  the  patrol  route. 

3.  After  the  Experiment 

At  the  end  of  each  session,  a  participant  removed  the  headphones,  stood 
up,  and  the  research  team  escorted  them  out  of  the  testing  area.  A  member  of 
the  team  downloaded  the  data  from  the  participant’s  session  and  Prepared  the 
simulation  for  the  next  participant. 
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After  the  final  session,  participants  answered  a  three-question  survey  in 
order  to  assess  their  opinion  of  the  training  and  gain  insights  into  the 
effectiveness  of  the  experiment.  The  research  team  thanked  participants  for  the 
four  weeks  of  dedication  to  the  process,  and  then  they  escorted  them  from  the 
testing  area. 

D.  MATERIALS 

1 .  Hardware 

In  order  to  minimize  the  negative  effects  of  virtual  3D  graphics  associated 
with  low  frame  rates  and  graphical  artifacts,  a  high-end  desktop  computer 
powered  the  experiment.  It  contained  the  following  specifications: 

•  Intel  17  930  @4.0  GHz  CPU 

•  G. SKILL  Ripjaws  Series  12GB  (6  x  2GB)  240-Pin  DDR3  SDRAM 
DDR3  2000 

•  2x  EVGA  Superclocked  Nvidia  GTX  480  GPU  PCI-Express  cards 
(SLI) 

•  Western  Digital  VelociRaptor  600GB  10000  RPM  32MB  Cache 
SATA  6.0Gb/s  HDD 

•  Microsoft  Sidewinder  X8  Black  12  Buttons  Tilt  Wheel  2.4GHz 
Wireless  Gaming  Mouse 

•  Logitech  Illuminated  Keyboard  104-key 

•  Plantronics  Gamecom  777  Gaming  Headphones 

Although  this  is  a  high-end  desktop  system  and  not  readily  deployable,  it 
resembles  what  will  be  available  with  future  distributable  technologies. 

Participants  used  five  keys  on  a  standard  104-key  PC  keyboard  during  the 
experiment.  The  post-experiment  survey  required  two  additional  keys.  The 
research  team  labeled  seven  keys  with  masking  tape  so  that  the  participants 
could  easily  identify  the  correct  keys.  Two  keys  commanded  the  vehicle  Al  to 
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proceed  to  the  next  waypoint.  Three  keys  correlated  to  confidence  levels  at 
identifying  changes.  The  last  two  keys  logged  yes  and  no  input  for  the  post¬ 
experiment  survey. 


Figure  1 1 .  Close  up  of  the  keyboard  configuration 


The  team  decided  to  utilize  a  very  large  screen  monitor,  28-inches 
diagonally,  for  a  more  immersive  and  comfortable  environment.  Noise-cancelling 
gaming  headphones  increased  immersion  for  the  audio  modality.  These 
headphones  also  blocked  out  other  noise  in  the  lab  and  allowed  participants  to 
focus  on  the  task  of  change  detection.  All  participants  tested  on  the  same 
hardware,  with  the  labeled  104-key  keyboard,  headphones,  and  mouse. 
Participants  sat  with  their  face  approximately  24  inches  from  the  screen 
throughout  the  experiment.  Due  to  family  emergencies,  two  participants  used  a 
slightly  smaller  26-inch  display  for  one  session  each. 
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2. 


Data  Collection  Systems  and  Software 


VBS2  wrote  participant  data  to  several  text  files  during  the  simulation 
using  the  external  script  files.  The  research  gathered  the  following  key  points  of 
data  anytime  a  participant  pressed  a  key  to  attempt  to  identify  a  change: 

•  direction  of  head  relative  to  the  vehicle  in  degrees 

•  simulation  time  in  hundredths  of  a  second 

•  confidence  level  key  pressed  (“H,”  “M,”  or  “L”) 

VBS2  copied  this  data  to  a  text  file  called  “clicked.”  Additionally,  any  time 
a  participant  pressed  a  button  to  identify  a  change,  the  scripts  incremented  tally 
variables  recording  the  number  of  clicks  and  confidence  level.  If  a  change 
existed,  when  and  where  the  participant  pressed  a  confidence  key,  VBS2 
incremented  the  correct  detection  tally  variable.  If  a  participant  correctly 
identified  a  change,  the  scripts  wrote  a  line  to  the  text  file  named  “targetsFound.” 
This  line  recorded: 

•  the  target  name 

•  the  simulation  time 

•  the  relative  head  direction  of  the  participant 

•  the  confidence  level  key  pressed. 

By  subtracting  the  number  of  detections  form  total  clicks,  the  research 
team  obtained  the  total  number  of  false  alarms  per  session.  During  the  second 
session  each  week,  the  false  alarms  were  the  total  number  of  clicks,  since  these 
were  “noise  only”  iterations. 

At  the  end  of  the  scenario,  VBS2  wrote  the  tally  variables’  values  to  a  text 
file  named  “targetFinalScore.”  To  facilitate  further  analysis  of  participant 
behavior,  the  VBS2  scripts  copied  a  participant’s  relative  head  direction  to  the 
vehicle  to  the  “headDirection”  text  file  every  200  milliseconds. 
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3.  Surveys 


Prior  to  testing,  each  participant  filled  out  a  pre-test,  demographic  survey. 
This  survey  identified  specific  characteristics  needed  for  blocking  and  randomly 
assigning  participants  into  experimental  groups  (Appendix  E). 

Following  the  final  experimental  session,  each  participant  answered  a 
three-question  automated  survey.  There  were  two  versions  of  the  survey,  one 
for  each  of  the  experimental  groups.  Participants  in  the  “training”  (experimental) 
group  answered  the  following  questions  in  the  simulation  using  labeled  keys: 

•  Question  1:  “After  completing  these  12  sessions  of  Change 

Detection  training,  do  you  feel  that  you  would  perform  better  at 
tasks  related  to  identifying  change  in  the  real  world?” 

•  Question  2:  “Do  you  feel  that  the  tutorial  lessons  you  received  at 
the  start  of  Weeks  1  and  7  helped  you  better  identify  change?” 

•  Question  3:  “Do  you  feel  it  was  easier  to  identify  changes  which 
indicated  a  high  level  threat  versus  changes  that  suggested  little  or 
no  threat?” 

Participants  in  the  “no  training”  (control)  group  answered  the  following 
questions  in  the  simulation  using  labeled  keys: 

•  Question  1:  “After  completing  these  12  sessions  of  Change 

Detection  training,  do  you  feel  that  you  would  perform  better  at 
tasks  related  to  identifying  change  in  the  real  world?” 

•  Question  2:  “Do  you  feel  that  you  would  have  performed  better  had 
you  received  instructional  training  as  part  of  this  experience?” 

•  Question  3:  “Do  you  feel  it  was  easier  to  identify  changes  which 
indicated  a  high  level  threat  versus  changes  that  suggested  little  or 
no  threat?” 
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IV.  RESULTS  AND  DISCUSSION 


A.  DEMOGRAPHICS 

Participants  in  this  experiment  came  from  the  student  body  at  NPS. 
Sixteen  volunteers  began  the  study;  however,  only  fifteen  completed  the 
experiment.  All  participants  were  male  between  the  ages  of  28-41 .  The  average 
age  was  34  years  old  with  a  standard  deviation  of  4.6  years. 


Catgory 

Specific  Category 

Total 

Training 

No  Training 

Navy 

6 

3 

3 

Army 

3 

2 

1 

Branch  of  Service 

Marines 

2 

0 

2 

Air  Force 

2 

1 

1 

International 

2 

1 

1 

04 

8 

4 

4 

Rank 

03 

6 

3 

3 

02 

1 

0 

1 

Play  Video  Games 

Yes 

9 

4 

5 

No 

6 

3 

3 

Gamers 

Yes 

7 

2 

5 

No 

8 

5 

3 

Ground  Combat 

Yes 

3 

2 

1 

Experience 

No 

12 

5 

7 

Table  6.  Demographic  data  by  group 

Table  6  shows  the  totals  for  each  demographic  category  and  the 
breakdown  of  participants  for  each  experimental  group.  The  group  labeled 
“Training”  was  the  experimental  group  that  received  the  short  virtual  tutor  prior  to 
Sessions  3  and  9.  The  group  labeled  “No  Training”  received  no  tutorials  for 
change  detection. 
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B.  ANALYSIS  OF  CORRECT  DETECTIONS  (HITS) 


When  a  participant  detected  a  change  in  the  virtual  environment,  they 
pressed  a  key  corresponding  with  high,  medium,  or  low  confidence.  When  the 
participant  pressed  any  confidence  key,  a  VBS2  script  logged  the  keystroke.  The 
first  output  of  interest  to  the  research  team  was  the  number  of  correct  detections 
by  session.  Sessions  3,  6,  9,  and  12  contained  actual  changes.  Figure  12 
shows  the  improvement  across  all  groups  at  detecting  changes  in  those 
sessions. 

Average  Detections  By  Session 
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Figure  12.  Mean  detections  by  session 
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Correct  Detections:  Scenario  1  (Sessions  3  &  9) 


Legend 

[J]  Session  3:  Hits 
[j]  Session  9:  Hits 


Figure  13.  Correct  detections  (hits)  for  all  participants,  Scenario  1 


In  addition  to  looking  at  the  mean  detections  for  all  participants  across  all 
sessions,  the  research  team  also  analyzed  the  number  of  correct  detections 
within  the  two  scenarios  for  all  participants.  Figure  13  shows  the  improvement 
for  all  participants  from  Session  3  to  Session  9.  These  two  sessions  utilized 
Scenario  1 . 


Session  9:  Hits 

23.0667 

t-Ratio 

5.363804 

Session  3:  Hits 

15.4667 

DF 

14 

Mean  Difference 

7.6 

Prob  >  |t| 

<0001* 

Std  Error 

1.4169 

Prob  >  t 

<.0001* 

Upper  95% 

10.639 

Prob  <t 

1.0000 

Lower  95% 

4.56104 

N 

15 

Correlation 

0.27217 

Table  7.  JMP  output  for  correct  detections,  all  participants,  Scenario  1 

Table  7  shows  the  JMP  output  for  the  paired-t  test  conducted  on  this  data. 
The  average  improvement  for  all  participants  was  an  increase  in  7.6  correct 
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detections  out  of  thirty  (30)  possible  changes.  This  improvement  occurred 
across  Session  3  to  Session  9  with  a  standard  deviation  of  1.42.  This  had  a  p- 
value  less  than  0.0001 . 
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Correct  Detections:  Scenario  2  (Sessions  6  &  12) 
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Figure  14.  Correct  detections  (hits)  for  all  participants,  Scenario  2 


Conducting  the  same  analysis  of  Scenario  2,  the  research  team  analyzed 
the  number  of  correct  detections  within  Scenario  2  sessions  for  all  participants. 
Figure  14  shows  the  improvement  for  all  participants  from  Session  6  to  Session 
12.  These  two  sessions  utilized  Scenario  2. 


Session  12:  Hits 

23.1333 

t-Ratio 

5.104072 

Session  6:  Hits 

20.6 

DF 

14 

Mean  Difference 

2.53333 

Prob  >  |t| 

0.0002* 

Std  Error 

0.49634 

Prob  >t 

<0001* 

Upper  95% 

3.59787 

Prob  <t 

0.9999 

Lower  95% 

1.4688 

N 

15 

Correlation 

0.84881 

Table  8.  JMP  output  for  correct  detections,  all  participants,  Scenario  2 
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Table  8  shows  the  JMP  output  for  the  paired-t  test  conducted  on  this  data. 
The  average  participant  improved  by  2.5  correct  detections  out  of  a  possible 
thirty  (30)  changes.  This  improvement  occurred  from  Session  6  to  Session  12 
with  a  standard  deviation  of  0.5.  While  this  data  reveals  a  smaller  degree  of 
improvement,  it  also  contains  less  variance.  This  could  indicate  that  participants 
were  all  reaching  a  more  stable  ability  to  detect  changes  in  their  environment; 
however,  due  to  the  small  sample  size,  results  did  not  support  this  assumption. 

Finally,  the  team  analyzed  the  correct  detections  over  both  scenarios  and 
all  four  sessions.  Displaying  the  boxplots  in  sequence  for  all  four  sessions  looks 
similar  to  Figure  12  that  illustrated  the  average  number  of  detections  as  a  smooth 
curve  over  time.  The  boxplots  in  Figure  15  provide  some  indication  of  the 
variance  within  each  session. 


Correct  Detections:  Sessions  3.  6.  9. 12 


Legend 

Q  Session  3:  Hits 
Q  Session  6  Hits 
Q  Session  9:  Hits 
tjl  Session  12:  Hits 


Figure  15.  Correct  detections  (hits)  for  all  participants,  all  sessions 

Figure  15  revealed  a  steady  increase  in  detections  and  decrease  in 
variance  over  time.  The  exception  to  this  trend  was  Session  12,  the  final 
session,  where  the  variance  increased  again.  The  research  team  attributed  this 
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revelation  to  participants  “letting  down”  after  four  weeks  of  testing;  however, 
there  is  no  data  to  support  this  assumption.  This  also  suggests  that  nine  (9) 
sessions  are  enough  because  the  participants  display  little  improvement  after  the 
ninth  session.  Of  note,  some  participants  anecdotally  stated  that  the  length  of 
the  experiment  started  to  wear  them  down. 


a  Oneway  Anova 


zd  Summary  of  Fit 


Rsquare 

0.392483 

Adj  Rsquare 

0.359938 

Root  Mean  Square  Error 

4.013371 

Mean  of  Response 

20.56667 

Observations  (or  Sum  Wgts) 
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Analysis  of  Variance 

Sum  of 


Source 

DF 

Squares 

Mean  Square 

F  Ratio  Prob  >  F 

Session 

3 

582.7333 

194.244 

12.0595  <0001* 

Error 

56 

902.0000 

16.107 

C.  Total 

59 

1484.7333 

<4  Means  for  Oneway  Anova 


Level 

Number 

Mean 

Std  Error 

Lower  95% 

Upper  95% 

3 

15 

15.4667 

1.0362 

13.391 

17.543 

6 

15 

20.6000 

1.0362 

18.524 

22.676 

9 

15 

23.0667 

1.0362 

20.991 

25.143 

12 

15 

23.1333 

1.0362 

21.057 

25.209 

Std  Error  uses  a  pooled  estimate  of  error  variance 

zd  Means  and  Std  Deviations 


Std  Err 


Level 

Number 

Mean 

Std  Dev 

Mean 

Lower  95% 

Upper  95% 

3 

15 

15.4667 

5.31664 

1.3728 

12.522 

18.411 

6 

15 

20.6000 

3.43927 

0.8880 

18.695 

22.505 

9 

15 

23.0667 

3.43234 

0.8862 

21.166 

24.967 

12 

15 

23.1333 

3.54293 

0.9148 

21.171 

25.095 

Table  9.  JMP  ANOVA  for  correct  detections  over  all  sessions 


Table  9  shows  the  statistical  output  for  the  analysis  of  variance  (ANOVA) 
for  Sessions  3,  6,  9,  and  12.  The  resulting  p-value  less  than  0.0001  indicates 
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there  is  statistically  significant  improvement,  as  shown  by  the  paired  t-tests 
(Table  7  and  Table  8)  for  each  scenario.  Appendix  H  contains  comprehensive 
data  in  JMP  used  for  these  tests. 

The  results  of  this  research  supported  the  research  hypothesis  predicting 
detection  rates  would  increase  over  time.  In  fact,  the  research  team  expected  a 
result  exactly  like  the  one  in  this  experiment.  The  steep  increase  in  detections 
early  in  the  experiment  followed  by  asymptotic  performance  at  the  end  of  the 
experiment  resembled  a  traditional  learning  curve. 

Perhaps  the  only  unexpected  outcome  in  the  analysis  of  correct  detections 
was  how  significantly  participants  improved  over  the  first  two-weeks.  The 
research  team  assumed  that  participants  would  naturally  get  better  from  week-to- 
week;  however,  the  magnitude  of  improvement  was  a  welcome  surprise. 

C.  ANALYSIS  OF  FALSE  ALARMS  (FA) 

Each  week,  the  research  team  collected  false  alarm  data  during  two 
sessions.  The  simulation  recorded  false  alarm  data  for  the  second  session  each 
week.  During  that  session,  the  in-simulation  voice-over  instructed  a  participant  to 
search  for  changes  in  the  environment.  The  second  sessions  each  week  were 
Sessions  2,  5,  8,  and  1 1 .  Unbeknownst  to  participants,  there  were  no  changes  to 
the  environment  in  any  of  those  sessions.  This  data  provided  an  indication  of  a 
participant’s  propensity  to  press  a  button  for  change  and  provided  the  classic 
Signal  Detection  Theory  case  where  there  is  noise  without  a  signal.  This  data 
also  provided  insights  into  bias  as  described  by  SDT.  The  second  time  the 
research  team  collected  data  on  false  alarm  rates  was  during  the  third  session 
each  week  in  which  there  were  signals  among  the  noise,  as  previously  described 
for  correct  detections. 

As  was  the  case  with  correct  detections,  when  a  participant  thought  they 
detected  a  change  in  the  virtual  environment,  they  pressed  a  key  corresponding 
with  high,  medium,  or  low  confidence.  VBS2  scripts  logged  the  keystroke.  The 
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first  output  of  interest  to  the  research  team  was  the  number  of  false  alarms  when 
there  were  no  signals  in  the  noise.  Sessions  2,  5,  8,  and  11  contained  no 
changes  (signals).  Figure  16  shows  a  sharp  decay  of  false  alarm  averages 
across  all  participants  during  these  sessions. 


Figure  16.  Mean  false  alarms  by  “Noise  Only”  sessions 


Next,  the  research  team  looked  at  the  false  alarms  for  all  sessions.  This 
included  the  sessions  with  and  without  the  presence  of  signals  in  the  noise. 
Recall  that  Sessions  2,  5,  8,  and  11  were  noise  only.  Sessions  3,  6,  9,  and  12 
contained  signals,  or  changes,  in  the  noise.  Figure  17  depicts  false  alarms 
across  all  sessions 
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Figure  17.  False  alarms  over  all  sessions 


The  results  showed  a  tendency  for  more  false  alarms  when  there  are 
signals  in  the  environment.  Some  of  these  false  alarms  manifested  themselves 
in  the  data  due  to  participants  who  pressed  a  button  to  detect  a  change  when  the 
change  was  still  outside  the  allowable  15-degree  and  50-meter  range  from  the 
vehicle.  Other  false  alarms  registered  due  to  a  participant’s  inaccuracy  moving 
the  mouse  over  a  target.  In  both  these  cases,  a  participant  detected  the  change, 
but  due  to  participant  error,  VBS2  registered  a  false  alarm.  The  research  team 
tried  to  account  for  these  errors  as  much  as  possible;  however,  there  was  no 
objective  way  to  determine  user  error  versus  an  actual  false  alarm.  Granted, 
these  errors  accounted  for  only  a  portion  of  false  alarms  per  session;  however,  it 
makes  the  second  session  of  each  week  a  more  accurate  measure  of  false 
alarms.  During  those  sessions,  any  press  of  a  button  to  denote  a  change  in  the 
environment  was  an  actual  false  alarm.  This  method  also  aligns  itself  with  the 
more  traditional  Signal  Detection  Theory  methods  of  testing  where  the 
environment  is  not  continuous,  but  rather,  a  single  instance  contains  a  signal  or  it 
does  not. 
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Figure  18.  False  alarms  signal  or  no  signal  present 


Figure  18  separates  the  fluctuating  curve  of  Figure  17  into  two  distinct 
curves.  These  two  curves  distinguish  between  false  alarms  occurring  in  sessions 
with  noise  only  and  sessions  with  signals-plus-noise.  In  both  cases,  the  false 
alarm  rate  continues  to  decrease  over  time,  even  with  the  additional  false  alarms 
in  the  sessions  with  signals  in  the  noise 


66 


lt  Oneway  Analysis  of  FA:  Noise  Only  By  FA  Session 


FA  Session 


Excluded  Rows  4 


Figure  19.  Boxplot  of  false  alarms,  noise  only  sessions 


One  participant  had  129  false  alarms  in  Session  3.  Figure  19  excludes 
this  participant’s  data.  Of  note,  even  with  that  outlier,  the  result  was  significant; 
however,  excluding  his  data  actually  increased  the  significance  of  the  result.  The 
boxplots  in  Figure  19  showed  the  significant  decrease  in  participant  average 
false  alarms  over  time.  The  box  plots  also  demonstrated  the  continual  decrease 
in  variance  of  false  alarms  over  time. 
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a  Oneway  Anova 


a  Summary  of  Fit 


Rsquare 

0.490455 

Adj  Rsquare 

0.461058 

Root  Mean  Square  Error 

7.718609 

Mean  of  Response 

11.03571 

Observations  (or  Sum  Wgts) 

56 

Analysis  of  Variance 

Sum  of 


Source 

DF 

Squares 

Mean  Square 

F  Ratio  Prob  >  F 

FA  Session 

3 

2981.9286 

993.976 

16.6839  <0001* 

Error 

52 

3098.0000 

59.577 

C.  Total 

55 

6079.9286 

A  Means  for  Oneway  Anova 


Level 

Number 

Mean 

Std  Error 

Lower  95% 

Upper  95% 

2 

14 

22.7143 

2.0629 

18.57 

26.854 

5 

14 

11.5714 

2.0629 

7.43 

15.711 

8 

14 

5.8571 

2.0629 

1.72 

9.997 

11 

14 

4.0000 

2.0629 

-0.14 

8.139 

Std  Error  uses  a  pooled  estimate  of  error  variance 

a  Means  and  Std  Deviations 


Std  Err 


Level 

Number 

Mean 

Std  Dev 

Mean 

Lower  95% 

Upper  95% 

2 

14 

22.7143 

11.8933 

3.1786 

15.847 

29.581 

5 

14 

11.5714 

8.5998 

2.2984 

6.606 

16.537 

8 

14 

5.8571 

3.4386 

0.9190 

3.872 

7.843 

11 

14 

4.0000 

3.3282 

0.8895 

2.078 

5.922 

Table  10.  JMP  ANOVA  output  for  FA  during  noise  only  sessions  (N=14) 

Table  10  shows  the  statistical  output  for  the  ANOVA  for  Sessions  2,  5,  8, 
and  1 1 .  The  resulting  p-value  less  than  0.0001  demonstrates  there  is  statistically 
significant  reduction  in  false  alarms  over  time.  Appendix  H  contains 
comprehensive  tables  of  the  JMP  data  used  for  these  tests. 
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Oneway  Anova 

a  Summary  of  Fit 

~) 

Rsquare 

0.296853 

Adj  Rsquare 

0.259184 

Root  Mean  Square  Error 

16.08837 

Mean  of  Response 

13.38333 

Observations  (or  Sum  Wgts) 

60 

a  Analysis  of  Variance 


Sum  of 


Source 

DF 

Squares 

Mean  Square 

F  Ratio  Prob  >  F 

FA  Session 

3 

6119.383 

2039.79 

7.8807  0.0002* 

Error 

56 

14494.800 

258.84 

C.  Total 

59 

20614.183 

A  Means  for  Oneway  Anova 


Level 

Number 

Mean 

Std  Error 

Lower  95% 

Upper  95% 

2 

15 

29.8667 

4.1540 

21.55 

38.188 

5 

15 

13.2000 

4.1540 

4.88 

21.521 

8 

15 

6.5333 

4.1540 

-1.79 

14.855 

11 

15 

3.9333 

4.1540 

-4.39 

12.255 

Std  Error  uses  a  pooled  estimate  of  error  variance 

zd  Means  and  Std  Deviations 


Std  Err 


Level 

Number 

Mean 

Std  Dev 

Mean 

Lower  95% 

Upper  95% 

2 

15 

29.8667 

29.9782 

7.7403 

13.265 

46.468 

5 

15 

13.2000 

10.4143 

2.6890 

7.433 

18.967 

8 

15 

6.5333 

4.2235 

1.0905 

4.194 

8.872 

11 

15 

3.9333 

3.2175 

0.8308 

2.152 

5.715 

Table  1 1 .  JMP  ANOVA  for  FA  during  noise  only  sessions  (N=15) 


Table  1 1  shows  the  statistical  output  for  the  ANOVA  for  Sessions  2,  5,  8, 
and  11  using  all  participants’  data  (N=15).  The  resulting  p-value  is  0.0002. 
There  is  statistically  significant  reduction  in  false  alarms  over  time.  By  excluding 
one  outlier  (participant),  the  variance  decreases  from  29.97  to  11.89  for  session 
2. 

Participant  data  clearly  showed  a  sharp  decay  in  false  alarm  rates  over 
time.  This  result  supported  the  hypothesis  that  false  alarms  would  decrease  over 
time.  Due  to  extensive  reading  on  Signal  Detection  Theory  literature,  the 
research  team  had  less  confidence  in  this  prediction  than  the  prediction  about 


69 


increased  detection  rates.  The  literature  often  spoke  of  bias,  an  individual’s 
propensity  to  say  there  is  (or  is  not)  a  signal  among  noise  more  than  50%  of  the 
time  (Proctor  et  al.,  2008).  The  research  team  suspected  that  if  participants 
began  to  detect  more  changes,  they  might  also  record  more  false  alarms.  As  it 
turned  out,  the  false  alarm  rates  decreased  as  detection  rates  increased.  This 
result  was  significant,  and  it  clearly  showed  the  participant’s  sensitivity  to 
changes.  The  resulting  sensitivity  represented  a  “best-case”  scenario,  exhibiting 
a  precise  capability  that  any  change  detection  trainer  should  possess. 

D.  CORRECT  DETECTIONS  VS.  FALSE  ALARMS 

One  of  the  hypotheses  for  this  study  sought  to  examine  if  it  was  possible 
for  detection  rates  to  increase  while  simultaneously  decreasing  false  alarm  rates. 
A  participant  could  potentially  increase  both  rates,  indicating  that  the  training  did 
not  improve  sensitivity  to  changes  in  the  environment.  Figure  20  illustrates  the 
percentage  of  key  presses,  or  clicks,  expended  for  correct  detections  versus 
false  alarms  during  the  sessions  with  signals  plus  noise. 


Figure  20.  Detections  versus  false  alarm  percentages  over  time 


70 


There  would  be  an  even  greater  distance  between  the  percentage  of  false 
alarms  and  correct  detections  if  the  research  team  had  developed  a  better 
method  to  discern  between  an  actual  false  alarm  and  human  error  in  range 
estimation.  The  previous  section  of  this  report  addressed  this  issue;  however, 
Figure  20  showed  the  research  team  that  there  could  have  been  a  larger  gap,  if 
an  improved  measure  for  false  alarms  existed  during  signal  plus  noise  trials. 

The  team  determined  during  deliberate  analysis  of  the  virtual  environment 
that  there  were  approximately  5000  potential  locations  for  a  change  (Appendix 
L).  This  discrete  number  of  locations  allowed  the  team  to  apply  classic  SDT 
metrics  in  a  continuous  virtual  environment.  During  a  signal-plus-noise  trial  in 
SDT,  previous  work  classified  a  participant’s  response  as  a  hit  or  miss. 
Traditionally,  signal-plus-noise  trails  do  not  produce  false  alarm  responses. 
Discretizing  the  environment  enabled  the  application  of  SDT  metrics  to  a 
continuous  environment. 

Small  sample  size  and  a  single  training  evolution  per  participant  make  it 
difficult  to  construct  meaningful  receiver  operating  characteristic  (ROC)  curves 
for  this  experiment.  While  Figure  20  is  not  a  traditional  SDT  sensitivity  (d’) 
analysis  or  ROC  curve,  the  research  team  believes  that  it  sufficiently 
demonstrates  the  sensitivity  of  the  training,  even  with  any  participant’s 
unintentional  false  alarm  clicks. 

E.  CONFIDENCE 

Measuring  a  participant’s  confidence  over  time  was  the  next  area  of 
interest  for  the  research  team.  As  mentioned  previously,  every  time  a  participant 
thought  they  detected  a  change  in  their  environment,  a  corresponding  key  press 
indicated  their  level  of  confidence.  This  section  of  the  report  analyzed 
confidence  over  time,  specifically,  confidence  at  correctly  detecting  changes  in 
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the  environment.  The  research  team  investigated  the  participants  as  a  whole 
and  at  each  group  to  determine  if  participants  became  more  confident  in  their 
ability  as  the  experiment  progressed. 


All  Participant  (N=15)  Confidence  of  Correct  Detections  (Hits)  across  all  Signal  +  Noise  Sessions 


Session  3  Session  6  Session  9  Session  12 

Signal  ♦  Noise  Testing  Session 


Figure  21 .  Confidence  levels  for  all  participants,  signal  +  noise 

Figure  21  clearly  shows  a  large  number  of  high  confidence  detections  by 
Session  12  when  compared  to  the  number  of  high  confidence  detections  in 
Session  3. 
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-  Oneway  Analysis  of  High  Confidence  By  Session 


o 


Session 


Figure  22.  Boxplot  for  confidence  by  session  (N=1 5) 
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Oneway  Anova 

a  Summary  of  Fit 

Rsquare 

0.561158 

Adj  Rsquare 

0.537648 

Root  Mean  Square  Error 

4.673634 

Mean  of  Response 

15.33333 

Observations  (or  Sum  Wgts) 

60 

Analysis  of  Variance 


Sum  of 


Source 

DF 

Squares 

Mean  Square 

F  Ratio  Prob  >  F 

Session 

3 

1564.1333 

521.378 

23.8695  <0001* 

Error 

56 

1223.2000 

21.843 

C.  Total 

59 

2787.3333 

a  Means  for  Oneway  Anova 


Level 

Number 

Mean 

Std  Error 

Lower  95% 

Upper  95% 

3 

15 

7.0667 

1.2067 

4.649 

9.484 

6 

15 

15.1333 

1.2067 

12.716 

17.551 

9 

15 

19.4000 

1.2067 

16.983 

21.817 

12 

15 

19.7333 

1.2067 

17.316 

22.151 

Std  Error  uses  a  pooled  estimate  of  error  variance 

A  Means  and  Std  Deviations 


Std  Err 


Level 

Number 

Mean 

Std  Dev 

Mean 

Lower  95% 

Upper  95% 

3 

15 

7.0667 

4.47958 

1.1566 

4.586 

9.547 

6 

15 

15.1333 

4.77892 

1.2339 

12.487 

17.780 

9 

15 

19.4000 

4.37199 

1.1288 

16.979 

21.821 

12 

15 

19.7333 

5.03511 

1.3001 

16.945 

22.522 

Table  12.  JMP  output  on  ANOVA  for  confidence  by  session  (N=15) 


Only  one  (1)  participant  never  used  the  “low”  or  “medium”  confidence  key 
in  any  session.  For  the  final  two  (2)  sessions,  two  (2)  participants  used  only  the 
“high”  confidence  key,  and  two  (2)  other  participants  used  only  the  “high” 
confidence  key  exclusively  during  one  (1 )  session. 
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A  Oneway  Anova 


A  Summary  of  Fit 


Rsquare 

0.54233 

Adj  Rsquare 

0.512481 

Root  Mean  Square  Error 

4.520705 

Mean  of  Response 

14.28 

Observations  (or  Sum  Wgts) 

50 

Analysis  of  Variance 

Source  DF  Sum  of  Squares  Mean  Square  F  Ratio  Rrob  >  F 

Session  3  1113.9883  371.329  18.1897  <  0001* 

Error  46  940.0917  20.437 

C.  Total  49  2054.0800 


A  Means  for  Oneway  Anova 


Level 

Number 

Mean  Std  Error 

Lower  95% 

Upper  95% 

3 

13 

6.6923  1.2538 

4.169 

9.218 

8 

14 

14.9288  1.2082 

12.497 

17.361 

9 

11 

17.4545  1 .3830 

14.711 

20.198 

12 

12 

18.8333  1.3050 

18.206 

21.480 

Std  Error  uses  a  pooled  estimate  of  error  variance 

i  Means  and  Std  Deviations 

Level 

Number 

Mean  Std  Dev  Std  Err  Mean 

Lower  95% 

Upper  95% 

3 

13 

6.6923  4.71495 

1.3077 

3.843 

9.542 

8 

14 

14.9288  4.83056 

1.3071 

12.105 

17.752 

9 

11 

17.4545  3.11010 

0.9377 

15.385 

19.544 

12 

12 

18.8333  4.91442 

1.4187 

15.711 

21.956 

Table  13.  ANOVA  for  confidence  by  session,  excluding  “high”  only  responses 

Table  12  shows  the  ANOVA  for  confidence  using  all  15  participants’ 
responses.  There  is  a  significant  difference  between  at  least  two  of  the  sessions 
with  a  p-value  of  0.0001.  Excluding  the  participants  who  only  used  the  “high” 
confidence  key,  Table  13  shows  the  result  is  still  significant.  The  F-ratio  only 
drops  slightly  from  23.8  down  to  18.1.  This  is  still  has  a  p-value  below  0.0001. 
Appendix  I  shows  a  complete  breakdown  of  confidence  key  usage  by  participant 
for  each  session. 
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Figure  23.  Confidence  levels  for  “Training”  group,  signal  +  noise 


Confidence  among  participants  in  the  “Training”  group  continued  to  rise 
throughout  the  experiment  as  shown  in  Figure  23.  Confidence  in  the  “No 
Training”  group  also  increased  over  time;  however,  Figure  24  highlights  a  small 
decrease  in  in  high  confidence  detections  during  the  final  session. 


No  Training  Group  (N=8)  Confidence  of  Correct  Detections  (Hits)  across  all  Signal  +  Noise 

Sessions 


Session  3  Session  6  Session  9  Session  12 

Signal  +  Noise  Testing  Session 


Figure  24.  Confidence  levels  for  “No  Training”  group,  signal  +  noise 
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F. 


THREAT  LEVEL  EFFECTS 


When  analyzing  the  correct  detection  results  by  threat  level,  the  research 
team  concluded  that  a  participant  does  not  demonstrate  a  preference  or  an  ability 
to  detect  one  threat  level  above  others.  This  result  indicated  that  a  user  might 
not  distinguish  between  a  target’s  threat  level  when  identifying  changes.  The 
variance  in  the  subjective  SME  ratings  for  threat  levels  and  small  participant 
sample  (N=15)  prevent  the  research  team  from  making  any  valid  conclusions. 
Therefore,  the  team  must  retain  the  null  hypothesis  that  there  is  no  difference  in 
the  number  of  detections  based  on  threat  level.  The  results  for  this  research 
question  were  inconclusive. 


Threat  within  Soeosro 


rhiiwil  Medium  Higii 

Each  err<y  tar  is  constructed  using  1  standard  deviation  from  the  mean. 

Figure  25.  Distribution  of  mean  detections  by  threat  level 
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G. 


GROUPS 


Two  groups  of  participants  completed  the  experiment.  One  hypothesis 
proposed  by  the  research  team  was  that  participants  receiving  an  automated 
virtual  tutor  would  perform  better  at  a  change  detection  task  than  those  who 
received  no  training.  Participants  of  equivalent,  rank,  branch-of-service,  combat 
experience,  and  video  game  ability  were  assigned  to  each  group.  The  results 
were  surprising  in  that  both  groups  performed  almost  equally.  This  does  not 
necessary  say  that  training  did  not  have  an  effect  on  those  in  the  training  group, 
but  the  research  team  cannot  say  that  the  automated  tutor  helped  the 
participants  in  that  group  perform  better  as  a  group  when  compared  to  the  group 
that  did  not  receive  any  training. 


Figure  26.  Mean  detections  per  session  by  experimental  group 


Figure  22  shows  how  closely  the  two  groups  performed  throughout  all  the 
signal-plus-noise  sessions.  The  two  (2)  curves  differ  by  approximately  one  (1) 
detection  in  Session  12  and  two  (2)  detections  in  Session  9. 
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Figure  27.  Mean  false  alarms  per  session  by  experimental  group  (N=15) 


Figure  28.  Mean  false  alarms  per  session  by  experimental  group  (N=14) 


Similarly,  false  alarm  rates  were  almost  identical  as  well,  as  shown  by 
Figure  27  and  Figure  28.  Interestingly,  the  average  false  alarm  rate  for  the 
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“training”  group  started  higher  than  those  without  training.  This  was  due  to  one 
participant  logging  129-false  alarms  during  Session  2.  After  removing  his  data 
from  the  false  alarm  data,  the  training  group’s  average  false  alarms  look  similar 
to  the  no  training  group,  as  seen  in  Figure  28. 

The  similarity  of  experimental  group  performance  due  to  small  sample  size 
prevents  the  research  team  from  making  any  valid  conclusions  concerning  the 
effect  of  tutorial  training.  Therefore,  the  team  must  retain  the  null  hypothesis  that 
says  there  is  no  difference  in  the  performance  of  the  “training”  and  “no  training” 
groups. 

H.  OTHER  EXPLORATORY  QUESTIONS 

The  research  team  explored  the  possibility  that  selective  demographic 
characteristics  predict  change  detection  performance.  Specifically,  the  team 
looked  at  the  sample  population’s  age,  rank,  branch-of-service,  combat 
experience,  and  video  game  experience.  We  expected  that  combat  experience, 
branch-of-service,  and  video  game  experience  might  significantly  contribute  to 
improved  performance  at  a  change  detection  task  in  a  virtual  environment.  For 
obvious  reasons,  the  team  thought  that  participants  with  ground  combat 
experience  would  be  better  at  detecting  change.  The  team  also  thought 
participants  who  played  video  games  would  perform  better  in  this  particular 
virtual  environment,  since  VBS2  is  a  Department  of  Defense  (DoD)  version  of  the 
commercial  game  Armed  Assault™. 

Using  the  data  included  in  Appendix  H,  the  team  used  JMP  to  determine  if 
any  factors  contributed  to  better  performance.  JMP  evaluated  all  data  points  for 
significance.  The  only  factor  that  showed  any  significance  was  branch-of- 
service.  Curiously,  Navy  and  Air  Force  were  the  predictor  variables  that  JMP 
identified  with  a  low  p-value  of  approximately  0.05.  Based  on  the  experience  of 
the  research  team,  this  prediction  was  a  surprise.  As  with  threat  level  and  group 
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assignment,  it  appeared  that  the  small  sample  size  produced  inconclusive 
results.  Future  studies  with  larger  sample  sizes  are  required  to  gain  confidence 
in  the  results. 

I.  SCENARIOS 

In  the  first  pilot  study,  the  research  team  suspected  that  at  least  one  of  the 
scenarios  was  more  difficult  than  the  other  scenarios.  Therefore,  there  was 
considerable  effort  to  ensure  that  the  two  scenarios  would  be  equally  difficult  for 
the  thesis  experiment.  To  validate  this  goal,  the  team  plotted  the  results  of  both 
scenarios  independent  of  each  other  for  both  iterations. 


Figure  29.  Average  detections  by  session  for  each  scenario 


Clearly,  there  is  improvement  between  the  first  session  of  Scenario  1  and 
the  first  session  of  Scenario  2.  This  report  outlined  the  significance  in  the  correct 
detections  portion  of  this  thesis.  However,  by  the  second  time  a  participant  saw 
each  scenario,  the  results  were  almost  identical.  As  seen  in  the  correct 
detections  analysis,  it  appears  that  the  average  number  of  correct  detections 
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approaches  an  asymptote  for  all  participants.  That  asymptote  appears  to  be  the 
same  regardless  of  scenario. 

J.  EXIT  SURVEY 

The  results  of  the  exit  survey  (Appendix  H)  revealed  that  14  of  the  15 
participants  felt  they  would  perform  better  at  change  detection  in  the  real  world. 
Although  subjective,  this  was  an  encouraging  result  for  the  validity  of  change 
detection  training  in  a  virtual  environment.  The  same  ratio  of  individuals  also  felt 
it  was  easier  to  detect  more  threatening  changes.  This  is  not  supported  by  the 
data  for  threat  level;  however,  participants  reported  high  confidence  in  their  ability 
to  detect  high  threat  changes.  Of  note,  the  dissenters  in  question  1  and  question 
3  were  not  the  same  participant. 


# 

Question 

Yes 

No 

1 

After  completing  these  12  sessions  of 
Change  Detection  Training,  do  you  feel  that 
you  would  perform  better  at  tasks  related  to 
identifying  change  in  the  real  world? 

14 

1 

2 

Training 

Do  you  feel  that  the  tutorial  lessons  you 
received  at  the  start  of  Weeks  1  and  7 
helped  you  better  identify  change? 

6 

1 

2 

No 

Training 

Do  you  feel  that  you  would  have  performed 
better  had  you  received  instructional 
training  as  part  of  this  experience? 

7 

1 

3 

Do  you  feel  that  it  was  easier  to  identify 
changes  which  indicated  a  high  level  threat 
vs.  changes  that  suggested  little  or  no 
threat? 

14 

1 

J 

Table  14.  Exit  survey  results 

Table  12  shows  the  slight  differences  in  question  2  offered  to  each  of  the 
groups.  In  the  “training”  group,  six  of  seven  participants  believed  the  training  was 
beneficial.  All  but  one  participant  in  the  “no  training”  group  expressed  that 
believed  they  would  perform  better  with  training.  This  was  an  interesting 
revelation,  as  the  data  showed  no  impact  of  the  training  on  the  change  detection 
ability  of  the  “training”  group. 
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V.  CONCLUSIONS  AND  RECOMMENDATIONS 


A.  OVERVIEW 

A  soldier’s  ability  to  detect  changes  in  their  environment  is  an  invaluable 
skill  on  the  battlefield.  The  first  one-hundred  days  of  combat  present  the  most 
threat  to  warfighters  due  to  unfamiliarity  with  the  terrain,  people,  patterns,  culture, 
attitudes,  and  beliefs  in  operational  areas.  Designing,  developing,  and  validating 
a  training  method  to  mitigate  the  dangers  presented  in  the  first  one-hundred  days 
is  a  necessary  research  endeavor. 

The  purpose  of  this  thesis  was  to  develop  and  demonstrate  a  prototype 
change  detection  trainer.  Ideally,  the  trainer  would  be  easily  distributable  to  DoD 
personnel,  convenient,  and  fun.  By  using  a  video  game  for  training,  the  research 
team  hoped  to  create  a  method  that  was  both  engaging  and  productive.  The 
team  defined  the  problem  as  “can  a  desktop  virtual  trainer  improve  change 
detection  skills?” 

B.  ANSWERS  TO  RESEARCH  QUESTIONS 

In  order  to  answer  the  thesis’s  problem  statement,  we  developed  these 
research  questions.  The  research  team  is  now  prepared  to  answer  each  of 
them: 

1 .  Does  change  recognition  performance  improve  over  time  in  a  virtual 
environment?  How  does  it  improve? 

Yes,  the  data  shows  that  performance  does  improve  over  time  in  a  virtual 
environment.  Participants  improved  at  the  number  of  changes  detected,  and 
they  reduced  the  number  of  false  alarms  they  committed  over  time. 
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a.  Can  a  participant  improve  at  recognizing  that  something  has 
changed  in  general,  measured  by  correct  detections  or  detection 
percentages? 

Yes,  participants  showed  statistically  significant  improvement  in  the 
number  of  correct  detections  in  signal  plus  noise  scenarios. 

b.  Does  a  participant’s  confidence  level  improve  at  recognizing  and 
identifying  changes  in  a  scene  using  a  simple  ordinal  scale  of  high, 
medium,  and  low? 

Yes,  the  data  supports  a  significant  improvement  in  user 
confidence  over  time.  However,  other  factors,  such  as  user  apathy,  could 
have  confounded  these  results. 

2.  How  do  detection  percentages  differ  when  assessing  a  participant’s 
performance  on  a  variety  of  threat  difficulty  levels? 

There  is  no  significant  statistical  support  to  argue  that  the  detection 
percentages  are  any  different  when  grouped  by  threat  level.  These  results  could 
be  skewed  by  SME  subjectivity  when  rating  the  threats,  or  the  way  participants 
perceive  the  threats  in  the  virtual  environment. 

3.  How  does  automated  virtual  training  affect  change  detection 
percentages? 

This  experiment  showed  no  effect  of  automated  virtual  tutors  on  change 
detection  training.  Larger  sample  sizes  would  produce  more  confidence  in  this 
result.  Additionally,  the  short  six  (6)  minute  tutorial  was  possibly  not  long  enough 
to  demonstrate  significant  results. 

4.  Does  the  percentage  of  recognized  changes  and  false  alarms  grow 
proportionally?  Alternatively,  can  the  detection  percentage  improve  while 
simultaneously  decreasing  the  false  alarm  rate  through  repeated  exposure  to  a 
simulated  environment? 
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Yes,  this  research  produced  significant  evidence  that  detection 
percentages  increase  while  false  alarm  rates  simultaneously  decrease  in  a  virtual 
environment. 

C.  OVERALL  ASSESSMENT 

There  are  numerous  positive  outcomes  from  this  research  effort.  An 
analysis  of  the  data  reveals  that  all  participants  improved  their  change  detection 
skills  over  the  course  of  four  weeks.  The  results  showed  that  detections 
increased  while  false  alarm  rates  decreased.  This  outcome  indicates  that 
participants  became  more  sensitive  to  changes  in  their  environment  over  time. 
The  results  also  indicate  a  positive  training  effect  within  the  virtual  environment. 

However,  not  all  results  turned  out  as  expected.  The  effects  of  the 
additional  tutorial  training  on  performance  were  negligible.  There  was  little  data 
to  support  any  effect  of  threat-level  on  a  participant’s  recognition  of  change. 

There  are  several  possible  hypotheses  for  why  these  unexpected  results 
occurred  that  are  covered  in  the  next  section;  however,  there  is  no  definitive  way 
to  explain  them  with  certainty.  Future  experimentation  should  investigate  these 
issues  and  build  upon  the  successful  prototype  design  for  training  change 
detection. 

D.  LIMITATIONS 

1.  Sample  Population 

This  study  enlisted  fifteen  volunteers  who  participated  consistently  over 
four  weeks.  Mid-grade  officers  comprised  a  majority  of  the  sample  population. 
Additionally,  all  participants  were  students  at  the  Naval  Postgraduate  School. 
While  it  is  important  for  all  soldiers  and  Marines  to  have  change  detection  skills,  a 
more  appropriate  sample  would  have  represented  young,  enlisted  combat-arms 
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soldiers.  These  personnel  would  normally  occupy  a  gunner  position  in  a 
HMMWV  or  patrol  on  foot,  constantly  looking  for  threat  indicators  or  changes  to 
their  environment. 

2.  Time 

A  second  limitation  on  this  study  was  the  time  available.  This  experiment 
lasted  six-weeks,  and  each  participant  devoted  four  weeks  of  their  time  to 
training  and  testing.  The  research  team  wished  that  testing  could  have  continued 
over  six  weeks  and  included  additional  sessions  in  order  to  gather  more  data. 
Limitations  on  both  the  research  team  and  participant’s  time  prohibited  such  a 
study.  This  time  limitation  prevented  the  team  from  recruiting  more  participants, 
enlisting  a  more  representative  sample,  expanding  the  experiment  over  time,  and 
evaluating  the  transfer  of  these  skills  to  the  real  world.  However,  the  results 
show  that  fewer  sessions  are  necessary  to  demonstrate  significant  improvement. 

With  respect  to  the  tutorial  training  sessions,  the  duration  of  six  (6) 
minutes  was  too  short  to  produce  significant  differences  between  groups. 
Allocating  a  greater  proportion  of  training  time  to  change  detection  tutorials  might 
produce  significant  results. 

3.  Tasks 

The  research  team  limited  participants  to  learning  the  virtual  environment 
during  baseline  sessions  and  looking  for  changes  during  testing  sessions.  There 
were  small  sub-tasks  such  as  moving  the  mouse  to  turn  the  head  and  pressing 
key  combinations  to  turn  the  vehicle  onto  the  next  waypoint.  The  literature 
discusses  the  use  of  secondary  tasks  that  can  distract  users  and  divide  attention. 
While  these  types  of  tasks  are  realistic,  the  research  team  did  not  feel  they  were 
appropriate  given  the  requirement  to  express  confidence  in  the  detected 
changes. 
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E.  RECOMMENDATIONS 

1.  Scenario  Recommendations 

Future  work  using  similar  scenario  designs  should  strive  to  improve  the 
methods  used  in  this  thesis.  VBS2  provides  the  capability  for  future  researchers 
to  immediately  expand  and  improve  the  scenarios  and  scripts  used  in  this  study. 

Scenario  development  should  focus  on  comparing  the  effects  of  shorter 
session  times.  The  duration  of  an  individual  session  or  the  number  of  total 
sessions  might  produce  revealing  results,  perhaps  optimizing  the  time  to  train  in 
a  virtual  environment  for  change  detection  tasks.  Resulting  data  might  alter 
when  the  user  has  reaches  an  asymptote  in  terms  of  performance. 

The  research  team’s  scenario  used  existing  VBS2  terrain  and  buildings. 
Using  existing  virtual  terrain  had  the  drawback  of  many  repeated  patterns 
through  the  scenario,  and  may  have  resulted  in  confounding  effects  over  several 
sessions  because  repeated  features  were  easy  to  identify  as  “noise”  in  the 
environment.  A  future  scenario  design  would  better  replicate  a  real  environment 
by  creating  few  or  no  repeating  structures.  With  the  recent  VBS2  1 .50  update, 
existing  buildings  and  other  objects  on  the  terrain  constructed  by  Bohemia 
Interactive  teams  can  be  moved  and  replaced.  This  capability  would  enable 
creation  of  an  environment  with  unique,  non-repeating  features. 

The  scenario  used  in  this  thesis  had  an  automated  system  for  identifying 
change.  This  method  made  it  more  difficult  to  assess  the  users’  intent  when  they 
pressed  a  key  to  identify  a  change.  Future  work  could  better  assess  user  intent 
by  asking  participants  to  verbalize  change  detections  to  an  observer.  Another 
possibility  is  to  create  a  system  where  the  simulation  pauses  while  the  user 
informs  the  system  of  their  intent,  leveraging  speech  recognition  software. 

Instead  of  repeating  the  same  scenario  each  week,  researchers  could 
create  a  pre  and  post-training  scenario  using  a  completely  new  scenario  and 
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geographical  area  each.  Assess  whether  the  same  reduction  of  false  alarms  and 
increase  in  correct  detections  exists  despite  a  complete  change  of  scenario. 

2.  Scoring  Recommendations 

The  scoring  methods  used  in  this  research  accomplished  the  goals  of 
logging  detections  and  false  alarms  in  the  appropriate  sessions.  Developing  a 
better  scoring  system  that  left  no  doubt  about  false  alarms  during  the  signal-plus- 
noise  sessions  would  reveal  the  true  number  of  false  alarms  in  a  continuous 
environment.  This  research  team  tested  a  method  to  accomplish  that  end  during 
pilot  studies,  but  it  required  a  participant  to  verbalize  their  detections.  While  this 
method  was  foolproof  for  knowing  the  true  number  of  false  alarms,  it  would  not 
allow  a  soldier  to  practice  these  scenarios  independently.  Therefore, 
verbalization  of  detections  did  not  meet  the  intent  of  the  prototype.  The  method 
utilized  in  this  experiment  was  currently  the  best  possible  way  to  employ  the 
categories  of  Signal  Detection  Theory  in  VBS2. 

3.  Recommendations  for  Future  Work 
a.  Sample  Population 

Using  this  experimental  design  and  scenario,  future  research 
should  strive  to  obtain  a  sample  population  from  Initial  Entry  Training  (IET) 
soldiers.  These  are  the  soldiers  that  will  occupy  the  front  line,  searching  for 
change.  Comparing  a  younger,  enlisted  population  to  the  older,  officer 
population  may  prove  to  be  an  interesting  comparison  of  different  populations. 

Given  more  time  and  resources,  future  work  with  an  IET  population 
could  test  these  scenarios  on  a  much  larger  number  of  participants.  A  larger 
sample  may  reveal  insights  undiscovered  by  this  research.  It  might  create  the 
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condition  where  the  automated  virtual  tutorial  shows  an  ability  to  accelerate  skill 
acquisition.  Using  an  intelligent  tutoring  system  (ITS)  could  also  improve 
performance. 


b.  Transfer  Study 

One  thing  the  research  team  cannot  know  is  whether  the  prototype 
enables  the  effective  transfer  of  change  detection  skills  to  real  world  conditions. 
This  is  an  important  next  step  for  the  trainer.  Again,  a  population  of  IET  soldiers 
who  are  about  to  undergo  IED  detection  training  as  part  of  their  Basic  Training 
would  be  a  perfect  population  to  test.  Allowing  a  sample  population  to  use  the 
prototype  trainer  developed  by  the  research  team  and  then  negotiate  a  live 
detection  course  would  be  ideal.  The  results  might  better  indicate  whether  the 
change  detection  trainer  accomplished  its  mission. 

c.  Comparing  Different  Training  Methods 

Another  approach  for  future  work  is  to  compare  change  detection 
training  in  virtual  environments  to  training  that  utilizes  still  photographs  or  full- 
motion  video.  Designing  an  experiment  where  participants  were  able  to  do  all 
three  methods,  VE,  photos,  and  video,  might  reveal  the  benefits  of  each.  Future 
training  developers  could  create  training  packages  for  change  detection  that 
incorporated  the  best  parts  of  each  genre.  A  full  curriculum  focused  on  change 
detection  training  would  likely  be  more  beneficial  than  any  one  of  these  elements 
used  independently. 

d.  Other  Distributable  Methods 

Developing  web  based  implementations  of  change  detection 
training,  possibly  using  Flash  or  HTML5  to  allow  distributed  learning  in  a  virtual 
environment  has  unlimited  potential.  Creating  a  operationally  focused  change 
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detection  “app”  for  mobile  devices,  such  as  a  tablet  or  phone,  would  be  extremely 
useful.  This  type  of  training  would  be  more  useful  and  accessible  to  every 
soldier. 


e.  Expanded  Training  Cognitive  Skills  Training 

Approached 

In  the  introduction  to  methodology,  the  research  team  identified 
eight  (8)  primary  tasks.  These  tasks  all  required  improved  cognitive  abilities. 
Expanding  change  detection  skill  training  to  include  memory  training,  such  as 
remembering  HVT  faces,  recalling  BOLO  vehicles,  and  remembering  CCIR 
enhances  readiness  for  the  COE. 

f.  Other  Devices 

The  objective  of  the  research  team  was  to  make  a  change 
detection  trainer  that  could  have  an  immediate  impact  on  training  within  the  Army 
or  Marine  Corps.  To  meet  that  end,  a  heavy  reliance  on  uncommon  input 
devices  or  output  devices  defeated  the  purpose.  The  tool  described  in  this  thesis 
represented  a  training  device  that  a  soldier  could  use  with  any  computer  and  a 
mouse.  However,  there  is  much  potential  for  future  work  using  head  mounted 
displays,  head  trackers,  and  eye  tracking.  In  addition  to  these  technologies, 
using  voice  recognition  input  devices  for  detecting  targets  might  be  one  way  to 
eliminate  erroneous  false  alarms.  The  potential  for  future  exploration  using  a 
multitude  of  input  and  output  devices  leaves  the  area  of  change  detection  in  a 
virtual  environment  an  interesting  and  worthy  pursuit. 
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APPENDIX  A.  PRE-EXPERIMENT  INSTRUCTIONS  AND 

MATERIALS 


A.  MAP 

Every  participant  received  a  copy  of  this  map  prior  to  the  experiment.  This 
map  was  also  available  to  every  participant  during  experimentation. 


Figure  30.  Overhead  map  provided  to  each  participant 
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B.  NO  TRAINING  GROUP 


Welcome  to  Change  direction  training. 

This  is  a  four-week  long  experiment.  Each  week  will  follow  the  same  procedure. 

Each  week  you  will  participate  in  one  15-minute  session  on  day  one,  a  15-minute  session  on  day 
two,  and  a  final  15-minute  session  on  day  three. 

In  order  to  save  time,  the  scenario  will  be  ready  to  start  when  you  arrive  to  the  testing  location. 
As  soon  as  you  take  your  seat,  you  will  be  allowed  to  start  the  session. 

Your  role  is  the  gunner  for  a  HMMWV  on  patrol  in  a  Middle-Eastern  city.  Attached  to  this 
document  is  a  map  that  you  can  use  to  orient  yourself  to  the  patrol  route. 

For  the  most  part,  navigation  will  be  handled  by  an  automated  driver.  However,  when  the  vehicle 
comes  to  a  stop,  it  is  up  to  you  to  order  the  vehicle  to  proceed  to  the  next  waypoint  by  pushing 

the  standard  keyboard  number  keys  1  and  then  9. 

Moving  the  mouse  will  allow  you  to  slew  your  gun  turret  and  thereby  change  your  view. 

Your  task  for  the  day  one  session  is  to  become  as  familiar  as  possible  with  the  presented  virtual 
city  area  .  During  the  first  session,  you  will  only  be  viewing  the  environment  by  moving  the 
mouse.  Make  “mental  notes”  and  try  to  learn  the  environment.  You  will  not  be  required  to  identify 
any  changes  in  the  first  session. 

Your  task  for  the  second  and  third  sessions  are  different  from  the  first  session.  You  are  to  identify 
changes  in  the  second  and  third  session  environments  by  pressing  labeled  keys  corresponding  to 
your  confidence  level  that  something  has  changed  from  what  you  saw  in  the  first  session 
environment.  (H-most  confident  to  L-least  confident) 

In  order  for  the  system  to  correctly  register  your  identification  of  a  change,  the  target  must  be 


Other  than  the  changes,  you  are  always  following  the  same  patrol  route  through  the  same  city 
streets.  Learn  your  environment  in  the  first  session  of  each  week  to  become  better  at  recognizing 
the  potential  changes  in  the  second  and  third  session. 


You  will  not  be  presented  with  your  scores  until  after  the  entire  experiment  is  complete.  However, 
you  will  hear  an  audible  tone  when  you  have  correctly  identified  a  change  during  the  second  and 
third  sessions. 
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c. 


TRAINING  GROUP 


Welcome  to  Change  direction  training. 

This  is  a  four-week  long  experiment.  Each  week  will  follow  the  same  procedure. 

Each  week  you  will  participate  in  one  20-minute  session  on  day  one,  a  15-minute  session  on  day 
two,  and  a  final  15-minute  session  on  day  three. 

In  order  to  save  time,  the  scenario  will  be  ready  to  start  when  you  arrive  to  the  testing  location. 
As  soon  as  you  take  your  seat,  you  will  be  allowed  to  start  the  session. 

Your  role  is  the  gunner  for  a  HMMWV  on  patrol  in  a  Middle-Eastern  city.  Attached  to  this 
document  is  a  map  that  you  can  use  to  orient  yourself  to  the  patrol  route. 

For  the  most  part,  navigation  will  be  handled  by  an  automated  driver.  However,  when  the  vehicle 
comes  to  a  stop,  it  is  up  to  you  to  order  the  vehicle  to  proceed  to  the  next  waypoint  by  pushing 

the  standard  keyboard  number  keys  1  and  then  9. 

Moving  the  mouse  will  allow  you  to  slew  your  gun  turret  and  thereby  change  your  view. 

Your  task  for  the  day  one  session  is  to  become  as  familiar  as  possible  with  the  presented  virtual 
city  area  .  During  the  first  session,  you  will  only  be  viewing  the  environment  by  moving  the 
mouse.  Make  “mental  notes”  and  try  to  learn  the  environment.  You  will  not  be  required  to  identify 
any  changes  in  the  first  session. 

Prior  to  this  first  session,  you  will  be  presented  with  a  short  tutorial  session. 

Your  task  for  the  second  and  third  sessions  are  different  from  the  first  session.  You  are  to  identify 
changes  in  the  second  and  third  session  environments  by  pressing  labeled  keys  corresponding  to 
your  confidence  level  that  something  has  changed  from  what  you  saw  in  the  first  session 
environment.  (H-most  confident  to  L-least  confident) 

In  order  for  the  system  to  correctly  register  your  identification  of  a  change,  the  target  must  be 
within  certain  parameters.  Specifically,  the  target  must  be  within  a  7.5  degree  arc  of  the  gun 


Other  than  the  changes,  you  are  always  following  the  same  patrol  route  through  the  same  city 
streets.  Learn  your  environment  in  the  first  session  of  each  week  to  become  better  at  recognizing 
the  potential  changes  in  the  second  and  third  session. 

You  will  not  be  presented  with  your  scores  until  after  the  entire  experiment  is  complete.  However, 
you  will  hear  an  audible  tone  when  you  have  correctly  identified  a  change  during  the  second  and 
third  sessions. 
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APPENDIX  B.  COMPUTER  SET-UP 


•  Install  VBS2  1 .31  or  better,  VTK  version  or  “Lite”  version. 

•  Label  the  “C,”  “V,”  and  “B”  keys  of  the  keyboard  “H,”  “M”  and  “L”  for 
ease  of  recognition  by  the  participant. 

•  From  the  archive  on  the  DVD,  extract  the  Change  Detection 
Training  folder  to  the  main  VBS2  install  folder’s  mission  folder;  this 
should  be  located  at  C:\Bohemia  lnteractive\VBS2\missions  if 
VBS2  is  installed  on  the  C  drive.  See  Figure  31 . 


Figure  31 .  File  view  for  training  scenarios 


•  The  mission  folders  in  .../missions  should  include: 
Session_1  .map_samawah50kmjeed.pbo 
Session_2  .map_samawah50km  Jeed .  pbo 
Session_3  .map_samawah50km_ieed .  pbo 
Session_4  .map_samawah50km_ieed .  pbo 
Session_5.map_samawah50km  Jeed. pbo 
Session_6  .map_samawah50km  Jeed .  pbo 
Session_7.map_samawah50km  Jeed.  pbo 
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Session_8.map_samawah50km_ieed.pbo 

Session_9.map_samawah50km_ieed.pbo 

Session_10.map_samawah50km_ieed.pbo 

Session_1 1  .map_samawah50km_ieed.pbo 

Session_12_T raining. map_samawah50km_ieed.pbo 

Session_12_Non_T raining.  map_samawah50km_ieed.pbo 

TrainingSession_1A.map_samawah50km_ieed.pbo 

TrainingSession_1B.map_samawah50km_ieed.pbo 

TrainingSession_2A.map_samawah50km_ieed.pbo 

Training2Session_2B.map_samawah50km_ieed.pbo 


Open  VBS2. 

Select  “Training  Scenarios”  from  the  main  menu. 

Select  “Change  Detection  Training”  and  click  “Open” 

Ensure  the  participant  is  wearing  headphones  or  using  speakers  at 
an  appropriate  volume  level  and  sitting  comfortably. 

For  the  first  session,  select  the  appropriate  scenario.  Click  “Start.” 

From  the  same  “File”  menu,  select  “Preview.” 

The  participant  should  follow  all  instructions  given  in  the  scenario 
which  features  voiceover  instructions. 

At  the  conclusion  of  the  scenario,  press  the  “ESC”  key.  Click 
“Abort.”  Now,  additional  scenarios  can  be  run  in  the  same  manner. 
However,  it  is  critical  that  any  data  files  be  extracted  as  follows  in 
the  next  step  BEFORE  running  another  scenario.  If  data  is  not 
extracted  before  running  another  scenario,  it  will  be  overwritten. 

All  scenarios,  except  the  training  scenarios,  will  write  to  data  files  in 
the  actual  VBS2  root  directory.  If  VBS2  was  installed  to  the  C  drive 
using  the  default  settings,  this  path  will  be  C:\Bohemia 
lnteractive\VBS2.  Table  15  shows  which  data  files  should  be 
obtained  from  this  directory  immediately  after  a  participant  has 
finished. 
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Scenario 

headDirection.txt 

startTimes.txt 

clicked.txt 

targetFinalScore.txt 

targetsFound.txt 

Survey.txt 

1,4,7,10 

V 

2,5,8,11 

V 

V 

V 

V 

3,6,9 

V 

V 

V 

V 

V 

12 

V 

V 

V 

V 

V 

V 

Training 

T able  1 5.  Data  file  matrix  for  training  scenarios 
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APPENDIX  C 


APPROVED  IRB  PROTOCOL 


Naval  Postgraduate  School 

Human  Research  Protection  Program 


From:  President,  Naval  Postgraduate  School 
Via:  Chairman,  Institutional  Review  Board 

To:  Dr.  Michael  McCauley,  MOVES  Institute 

Dr.  Quinn  Kennedy,  Operation  Research  Department 
Dr.  Anthony  Ciavarelli 
MAJ  Jason  Caldwell,  USA 
MAJ  Michael  Stinchfield,  USA 

Subj :  CHANGE  DETECTION:  STUDYING  THE  EFFECTS  OF  TRAINING  FREQUENCY 
AND  THREAT  SALIENCY  ON  PERFORMANCE  IN  A  VIRTUAL  ENVIRONMENT 

Enel:  (1)  Approved  IRB  Protocol 

1.  The  NPS  IRB  is  pleased  to  inform  you  that  the  NPS  Vice  President  and  Dean  of  Research  has 
approved  your  project  (NPS  IRB#  NPS.201 1.0064-IR-CONV-A).  The  approved  IRB  protocol  is 
found  in  enclosure  ( 1 ).  Completion  of  the  CITI  Research  Ethics  T  raining  has  been  confirmed. 

2.  This  approval  expires  on  30  September  1 1.  If  additional  time  is  required  to  complete  the 
research,  a  continuing  review  report  must  be  approved  by  the  IRB  and  NPS  President  prior  to  the 
expiration  of  approval.  At  expiration  all  research  (subject  recruitment,  data  collection,  analysis 
of  data  containing  PII)  must  cease. 

3.  Dr  Anthony  Ciavarelli  may  not  engage  in  research  activities  until  NPS  and  DON  HRPP  have 
approved  an  Individual  Investigator  Agreement  (IIA)  allowing  Dr.  Ciavarelli  coverage  under  the 
NPS  assurance  to  conduct  research  with  human  subjects.  The  IRB  Administrator  will  provide 
written  approval  to  the  research  team  once  received. 

4.  You  are  required  to  report  to  the  IRB  any  unanticipated  problems  or  serious  adverse  events  to 
the  NPS  IRB  within  24  hours  of  the  occurrence. 

5.  Any  proposed  changes  in  IRB  approved  research  must  be  reviewed  and  approved  by  the  NPS 
IRB  and  NPS  President  prior  to  implementation  except  where  necessary  to  eliminate  apparent 
immediate  hazards  to  research  participants  and  subjects. 

6.  As  the  Principal  Investigator  it  is  your  responsibility  to  ensure  that  the  research  and  the 
actions  of  all  project  personnel  involved  in  conducting  this  study  will  conform  with  the  IRB 
approved  protocol  and  IRB  requirements/policies. 
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7  After  the  experiment  is  completed  the  Principal  Investigator  will  submit  to  the  Human 
Subjects  Protection  Office,  all  signed  informed  consent  documents,  unanticipated  problem 
reports,  adverse  event  reports  and  a  End  of  Experiment  Report.  The  Human  Research  Protection 
I  rogram  Office  will  secure  these  documents  for  10  years  and  then  forward  to  the  nearest  FRC. 


Institutional  Review  Board  Naval  Postgraduate  School 
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APPENDIX  D.  INFORMED  CONSENT 


Change  Detection:  Studying  the  Effects  of  Training  Frequency  and  Threat 
Saliency  on  Performance  in  a  Virtual  Environment 

Informed  Consent  Form 

Introduction.  You  are  invited  to  participate  in  a  research  study  entitled:  “Change 
Detection:  Studying  the  Effects  of  Training  Frequency  and  Threat  Saliency  on 
Performance  in  a  Virtual  Environment 

Procedures.  This  study  requires  a  participant  to  view  a  virtual  environment  in  a 
“normal”  state,  and  then  subsequently  detect  changes  that  occur  in  the  same 
environment  over  the  course  of  four  (4)  weeks.  Participants  will  fill  out  a  short 
demographic  questionnaire  to  better  understand  the  user’s  military  experience 
and  level  of  computer  gaming  expertise.  The  survey  should  take  no  longer  than 
one  (1 )  minute  to  complete.  Assistance  will  be  given,  if  required.  During  the  first 
two  sessions,  there  will  be  an  overview  brief  that  explains  the  target  detection 
task  in  the  virtual  environment  and  the  participant’s  role  in  the  scenario.  There 
will  be  time  for  questions  after  the  brief.  The  briefing  will  take  no  longer  than  two 
(2)  minutes.  Users  will  then  complete  twelve  (12),  15-minute  scenarios  detecting 
as  many  changes  as  possible  in  a  simulated  environment.  Each  participant  will 
conduct  only  one  (1)  of  the  scenarios  in  a  single  day.  At  the  conclusion  of  the 
experiment  (all  12-scenarios),  users  will  be  released  from  the  study. 

Voluntary  Nature  of  the  Study.  Your  participation  in  this  study  is  strictly 
voluntary.  If  you  choose  to  participate  you  can  change  your  mind  at  any  time  and 
withdraw  from  the  study.  You  will  not  be  penalized  in  any  way  or  lose  any 
benefits  to  which  you  would  otherwise  be  entitled  if  you  choose  not  to  participate 
in  this  study  or  to  withdraw. 

Potential  Risks  and  Discomforts.  The  potential  risks  of  participating  in  this 
study  are  not  greater  than  minimal  risk.  There  is  the  possibility  of  a  participant 
experiencing  simulator  sickness  or  head  ache.  This  possibility  is  no  greater  than 
those  encountered  when  playing  civilian  video  games.  If  you  feel  nauseous  or 
dizzy,  inform  the  observers  and  they  will  escort  you  outside.  If  you  cannot 
complete  the  training,  it  will  be  rescheduled.  If  a  participant  continues  to 
experience  these  effects,  they  can  withdraw  from  the  study  at  any  time. 

There  is  the  additional  risk  of  experiencing  Post-Traumatic  Stress  Disorder 
(PTSD)  symptoms  due  to  the  detailed  nature  of  the  virtual  environment.  If  you 
have  experienced  the  symptoms  of  post  traumatic  stress  disorder  or  been 
treated  previously  for  post  traumatic  stress  disorder,  please  notify  the  data 
collector  and  remove  yourself  from  participation  in  this  study.  Possible 
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symptoms  include:  1 )  flashbacks,  2)  hallucinations,  3)  psychological  or 

physiological  distress,  4)  avoidance,  5)  lack  of  ability  to  recall  details  of  the 

event ,  6)  difficulty  falling  asleep ,  7)  irritability  or  outbursts  of  anger ,  8) 

difficulty  concentrating,  and  9)  being  easily  startled.  If  you  feel  PTSD-like 
stress,  notify  the  observers  and  you  will  be  removed  from  the  testing  area, 
provided  points  of  contact  numbers  and  withdrawn  from  the  study.  Medical 
support  for  those  experiencing  difficulties  is  available  at  831-242-4328  at  the 
Presidio  of  Monterey  Wellness  Center,  BLD  454. 

Anticipated  Benefits.  The  ability  to  detect  changes  in  an  environment  is  an 
important  skill  throughout  the  Department  of  Defense  (DoD).  By  showing  that 
virtual  environment  training  improves  change  recognition  skills,  we  demonstrate 
the  viability  of  VBS2  as  a  desktop  solution  for  soldiers  as  a  part-task  trainer. 
Since  all  U.S.  Army  soldiers  have  access  to  VBS2  for  their  personal  computers, 
our  research  could  make  change  detection  training  more  accessible  to  the 
individual  user.  This  training  could  reduce  the  risk  of  casualties  from  a  variety  of 
asymmetric  threats  in  the  first  one-hundred  days  in  combat  and  beyond. 

Compensation  for  Participation.  Participants  in  this  experiment  will  be  invited 
to  a  “Thank  You  Event”  that  will  include  food  and  drink,  such  as  a  pizza  party  or 
dinner  party,  to  occur  at  the  end  of  the  Spring  201 1  Quarter.  Additionally,  a  copy 
of  the  research  results  will  be  available  at  the  conclusion  of  the  experiment  by 
contacting  MAJ  Jason  Caldwell  or  MAJ  Mike  Stinchfield. 

Confidentiality  &  Privacy  Act.  Any  information  that  is  obtained  during  this 
study  will  be  kept  confidential  to  the  full  extent  permitted  by  law.  All  efforts,  within 
reason,  will  be  made  to  keep  your  personal  information  in  your  research  record 
confidential,  but  total  confidentiality  cannot  be  guaranteed. 

All  completed  surveys  will  be  sent  via  encrypted  and  digitally  signed  emails.  If  it 
is  necessary  to  print  any  surveys,  all  personally  identifiable  information  (Pll)  will 
be  scrubbed  so  the  participant  cannot  be  identified.  All  printed  surveys  will  be 
stored  in  a  locked  drawer  or  facility  when  not  in  use.  In  the  final  publication,  no 
names  will  be  used  for  privacy  and  because  it  adds  no  value  to  the  research. 

Points  of  Contact.  If  you  have  any  questions  or  comments  about  the  research, 
or  you  experience  an  injury  or  have  questions  about  any  discomforts  that  you 
experience  while  taking  part  in  this  study,  please  contact  the  Principal 
Investigator,  Dr.  Michael  McCauley,  memccaul@nps.edu.  Questions  about  your 
rights  as  a  research  subject  or  any  other  concerns  may  be  addressed  to  the 
Naval  Postgraduate  School  IRB  Chair,  CAPT  John  Schmidt,  USN,  831-656- 
3864,  jkschmid@nps.edu. 

Statement  of  Consent.  I  have  read  the  information  provided  above.  I  have 
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been  given  the  opportunity  to  ask  questions  and  all  the  questions  have  been 
answered  to  my  satisfaction.  I  have  been  provided  a  copy  of  this  form  for  my 
records  and  I  agree  to  participate  in  this  study.  I  understand  that  by  agreeing  to 
participate  in  this  research  and  signing  this  form,  I  do  not  waive  any  of  my  legal 
rights. 


Participant’s  Signature  Date 


Researcher’s  Signature  Date 

MAJ  Jason  C.  Caldwell 


Researcher’s  Signature  Date 

MAJ  Michael  K.  Stinchfield 


103 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


104 


APPENDIX  E. 


DEMOGRAPHIC  SURVEY 


1.  Please  enter  your  age:  _ years  old 

2.  Are  you  a  male  or  female?  (Circle  one)  MALE  FEMALE 


3.  Please  indicate  your  military  service  branch  or  select  civilian  (Circle  One). 

Army  Marine  Corps  Navy  Air  Force 

Civilian  Retired  Military  International  Military  Member 


4.  Please  indicate  your  military  grade  (eg.  0-3,  0-4,  E-6,  etc) _ - _ 

5.  Do  you  play  video  games  on  computers  (Do  NOT  count  consoles  -  Xbox,  PS3,  Wii)?  Circle  one: 

YES  NO 

6.  Circle  the  most  descriptive  unit  of  measurement  that  describes  the  amount  of  time  you  have 
played  computer-based  First  Person  Perspective  (First  Person  Shooters)  video  games: 

N/A  HOURS  DAYS  WEEKS  MONTHS  YEARS 

7.  Do  you  consider  yourself  a  gamer?  (Circle  One)  YES  NO 

8.  Have  you  conducted  urban  ground  combat  operations  involving  a  search  or  target/change 
detection  task?  (Example:  Neighborhood  patrol  in  Iraq/Afghanistan,  Route  clearance  operations, 
etc) 

YES  NO 

9.  Do  you  have  20/20  vision,  or  correctable  to  20/20  vision?  YES  NO 

10.  Are  you  color  blind?  YES  NO 

11.  Have  you  ever  experienced  Post-Traumatic  Stress  (PTS)  symptoms  that  might  be  aggravated 
by  seeing  computer  images  of  terrain  that  looks  like  Iraq  or  Afghanistan  (examples:  upsetting 
memories,  flashbacks,  nightmares,  feelings  of  intense  distress,  or  physical  reactions  -  pounding 
heart,  rapid  breathing,  nausea,  muscle  tension,  sweating)?  If  so,  inform  the  research  team  that 
you  will  be  unable  to  participate  in  this  study. 

12.  Have  you  ever  experienced  simulator  sickness?  YES  NO 

13.  Rate  your  agreement  with  the  following  statement:  "Computer  based  simulation  can  be  an 
effective  training  tool  for  unit  tactical  training." 

Strongly  Disagree 
Moderately  Disagree 
Mildly  Disagree 
No  Opinion 
Mildly  Agree 
Moderately  Agree 
Strongly  Agree 
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APPENDIX  F.  VIRTUAL  BATTLESPACE2  ™  SCRIPTS 


A.  INTRODUCTION 

External  script  file  code  bound  to  C,V,  and  B  keys  in  “init.sqf”  file, 
executes  when  bound  button  is  pressed. 

B.  KEY  BINDING  CODE 

1.  High  Confidence 

(“B”  keyboard  key  bound  to  lookat3.sqf) 

totalClicks  =  totalClicks  +  1; 
clickH  =  clickH  +  1 ; 

_3click  =  “highConf”; 

_blank  = 

_headDir  =  getDir  player; 

_vehDir  =  getDir  veh_1 ; 

_relDir  =  _headDir  -  _vehDir; 

_time  =  time; 

_filename  =  “targetsFound.txt”; 

_filename3  =  “clicked.txt”; 

_append 

format[“AppendLine(%1)@%2,%3,%4,”_filename3,_3click,_relDir,_time]; 
_writeFile  =  pluginFunction  [“VBSPIuginFileAccess”  ,  _append]; 

{ 


Code 
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if  ((player  isLookingAt  Lx.  1 5])  &&  ((player  distance  _x)  <  60))  then{ 
_filename  =  “targetsFound.txt”; 

_confidence  =  “confl  correct”; 
scoreH  =  scoreH  +  1 ; 
totalCorrect  =  totalCorrect  +  1 ; 

_append 

format[“AppendLine(%1)@%2,%3,%4,%5/Lfi|ename._x._re|Dir,_tirne,_3click]; 

_writeFile  =  pluginFunction  [“VBSPIuginFileAccess” ,  _append]; 
playSound  “dinger”; 

}; 

JforEach  myArray; 

2.  Mid  Confidence 

(“V”  keyboard  key  bound  to  lookat2.sqf) 

totalClicks  =  totalClicks  +  1; 
clickM  =  clickM  +  1 ; 

_2click  =  “midConf”; 

_blank  = 

_headDir  =  getDir  player; 

_vehDir  =  getDir  veh_1 ; 

_relDir  =  _headDir  -  _vehDir; 

_time  =  time; 

_filename  =  “targetsFound.txt”; 

_filename3  =  “clicked.txt”; 
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_append 

format[“AppendLine(%1)@%2,%3,%4,”_filename3,_2click,_relDir,_time]; 
_writeFile  =  pluginFunction  [“VBSPIuginFileAccess” ,  _append]; 

{ 

if  ((player  isLookingAt  |_x,  1 5])  &&  ((player  distance  _x)  <  60))  then{ 
_filename  =  “targetsFound.txt”; 

_confidence  =  “confl  correct”; 
scoreM  =  scoreM  +  1 ; 
totalCorrect  =  totalCorrect  +  1 ; 
playSound  “dinger”; 

_append 

format[‘AppendLine(%1)@%2,%3,%4,%5,”_filename,_x,_relDir,_time,_2click]; 

_writeFile  =  pluginFunction  [“VBSPIuginFileAccess” ,  _append]; 
playSound  “dinger”; 

}; 

}forEach  myArray; 

3.  Low  Confidence 

(“C”  keyboard  key  bound  to  lookatl.sqf) 

totalClicks  =  totalClicks  +  1; 
clickL  =  clickL  +  1 ; 

_1  click  =  “lowConf”; 

_blank  = 

_headDir  =  getDir  player; 
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_vehDir  =  getDir  veh_1 ; 

_relDir  =  _headDir  -  _vehDir; 

_time  =  time; 

_filename  =  “targetsFound.txt”; 

_filename3  =  “clicked.txt”; 

_append 

format[“AppendLine(%1  )@%2,%3,%4,”_filename3,_1  click, _relDir,_time]; 
_writeFile  =  pluginFunction  [“VBSPIuginFileAccess”  ,  _append]; 

{ 

if  ((player  isLookingAt  Lx.  1 5])  &&  ((player  distance  _x)  <  60))  then{ 
_filename  =  “targetsFound.txt”; 

_confidence  =  “confl  correct”; 
scoreL  =  scoreL  +  1 ; 
totalCorrect  =  totalCorrect  +  1 ; 

_append 

format[“AppendLine(%1  )@%2,%3,%4,%5,”_filename,_x,_relDir,_time,_1  click]; 

_writeFile  =  pluginFunction  [“VBSPIuginFileAccess”  ,  _append]; 
playSound  “dinger”; 

}; 

}forEach  myArray; 

C.  SCENARIO  INITIALIZATION  FILE  CODE 

This  code  edeuctes  the  “init.sqf”  file  when  the  scenario  begins 
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1. 


Session  1 


_blank  = 

_filename2  =  “headDirection.txt”; 

_append  =  format[“Write(%1)@%2,”_filename2,_blank]; 

_writeFile  =  pluginFunction  [“VBSPIuginFileAccess”  ,  _append]; 

hint  “Aim  at  the  Yellow  Cross.  This  is  the  maximum  distance  at  which  the 
system  will  allow  you  to  identify  changes.  The  Green  Arrows  represent  the 
maximum  allowable  azimuth  error.”; 

playSound  “tolerance”; 

sleep  23; 

gal  setPos  (getPos  remove); 
gar  setPos  (getPos  remove); 
yc  setPos  (getPos  remove); 

hint  “In  this  scenario  today,  you  will  not  be  identifying  changes.  You  are  to 
conduct  observation  as  the  gunner  during  the  patrol.  Move  your  mouse  to  rotate 
your  view.  Learn  this  environment.  In  the  next  two  sessions,  you  will  be  required 
to  identify  things  that  have  changed  from  the  environment  you  see  today.”; 

playSound  “introSceneOne”; 

sleep  25; 

hint  “Every  session’s  patrol  route  will  be  the  same.  It  will  last 
approximately  12  minutes.  We  will  follow  Route  Blue,  straight  ahead  of  you. 
Then,  we  will  make  a  right  turn  on  Route  Red.  Then,  when  we  reach  Route 
Green  at  Victory  Bridge,  we  will  take  another  right  and  follow  Route  Green  along 
the  river.  At  the  end  of  Route  Green,  we  will  make  a  right  turn  onto  Route  Purple 
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for  the  final  portion  of  our  patrol.  Again,  every  session’s  patrol  will  follow  this 
same  route.”; 

playSound  “route”; 

sleep  35; 

hint  “Again,  it  is  important  to  understand  that  today’s  session  exposes  you 
to  the  base  or  baseline  environment.  You  will  be  required  in  the  other  two 
sessions  of  this  week  to  identify  changes  from  this  environment  you  see  today. 
As  we  proceed  with  the  training,  you  will  see  this  baseline  environment  as  the 
first  session  of  every  week.”; 

playSound  “baseline”; 

sleep  25; 

hint  “Now,  press  1  then  9  to  start  your  patrol  up  Route  Blue”; 
playSound  “oneNine”; 
while  {alive  player}  do 
{ 

_headDir  =  getDir  player; 

_vehDir  =  getDir  veh_1 ; 

_relDir  =  _headDir  -  _vehDir; 

_time  =  time; 

_append  = 

format[“AppendLine(%1)@%2,%3,”_filename2,_relDir,_time]; 

_writeFile  =  pluginFunction  [“VBSPIuginFileAccess” ,  _append]; 

sleep  0.2; 

}; 
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2. 


Session  2 


0x2E  bindKey  “nul  =  []  execVM  ‘lookatl  .sqf”; 

0x2 F  bindKey  “nul  =  []  execVM  1ookat2.sqf”; 

0x30  bindKey  “nul  =  []  execVM  1ookat3.sqf”; 
myArray  =  [fake_truck]; 

_blank  = 
totalClicks  =  0; 
totalCorrect  =  0; 
scoreL  =  0; 
scoreM  =  0; 
scoreH  =  0; 
clickL  =  0; 
clickM  =  0; 
clickH  =  0; 

_filename  =  “targetsFound.txt”; 

_filename2  =  “headDirection.txt”; 

_filename3  =  “clicked.txt”; 

_filename4  =  “startTimes.txt”; 

_append  =  format[“Write(%1)@%2,”_filename,_blank]; 
_writeFile  =  pluginFunction  [“VBSPIuginFileAccess”  ,  _append]; 
_append  =  format[“Write(%1)@%2,”_filename2,_blank]; 
_writeFile  =  pluginFunction  [“VBSPIuginFileAccess”  ,  _append]; 
_append  =  format[“Write(%1)@%2,”_filename3,_blank]; 
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_writeFile  =  pluginFunction  [“VBSPIuginFileAccess”  ,  _append]; 

_append  =  format[“Write(%1)@%2,”_filename4,_blank]; 

_writeFile  =  pluginFunction  [“VBSPIuginFileAccess”  ,  _append]; 

hint  “Welcome  to  day  two  of  Change  Detection  Training.  In  your  last 
session,  you  were  allowed  to  observe  the  baseline  environment.  In  today’s 
session,  you  will  attempt  to  identify  any  changes  from  that  baseline  environment. 
Please  be  patient  and  listen  closely  as  I  share  some  important  instructions.”; 

playSound  “welcomeDayTwo”; 

sleep  22; 

hint  “To  identify  changes,  you  will  aim  the  gun  at  the  location  and  then 
press  the  key  labeled  H,  M,  or  L  according  to  your  confidence  level  that 
something  has  changed  from  the  baseline  session  -  High  Confidence,  Mid-Level 
Confidence,  Low  Confidence,  respectively.”; 

playSound  “conflevels”; 

sleep  20; 

hint  “Remember  from  the  instruction  sheet  you  received,  that  the  target 
you  want  to  identify  must  be  within  50  meters  of  your  location  and  within  a  15 
degree  cone  for  the  system  to  register  a  hit,  even  if  it  is  an  actual  change. 
Displayed  right  now  for  your  understanding  are  green  arrows  ahead  of  you  on  the 
road  displaying  the  limits  of  this  area.  If  you  try  to  identify  a  target  outside  these 
parameters,  it  will  not  register  as  a  hit  and  will  in  fact  register  as  a  miss.  Misses 
will  count  against  your  total  score.  It  is  best  to  wait  until  you  have  the  location  you 
want  to  target  reasonably  close  before  pressing  the  desired  key.”; 

playSound  “targeting”; 

sleep  40; 

hint  “I  will  now  remove  the  arrows”; 
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playSound  “removal”; 
sleep  4; 

gal  setPos  (getPos  remove); 
ga2  setPos  (getPos  remove); 
ga3  setPos  (getPos  remove); 
ga4  setPos  (getPos  remove); 
ga5  setPos  (getPos  remove); 
ga6  setPos  (getPos  remove); 
ga7  setPos  (getPos  remove); 
ga8  setPos  (getPos  remove); 
ga9  setPos  (getPos  remove); 
galO  setPos  (getPos  remove); 
gal  1  setPos  (getPos  remove); 
gal 2  setPos  (getPos  remove); 
yc  setPos  (getPos  remove); 
sleep  1 ; 

hint  “If  you  have  identified  an  actual  change  within  the  specified 
parameters,  you  will  hear  a  short  ding  confirming  the  hit.  If  you  press  a  button 
while  targeting  anything  else,  or  are  outside  the  parameters,  you  will  hear 
nothing.  Do  not  get  frustrated,  you  have  only  seen  the  baseline  once  so  far  in 
this  first  week  of  training.  Just  continue  to  learn  the  baseline  environment  better 
each  time  you  see  it  as  the  first  session  of  a  week.”; 

playSound  “dingExplanation”; 
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sleep  28; 

hint  “There  is  a  white  pick-up  truck  to  the  right  of  the  road  ahead  of  you. 
Go  ahead  and  aim  at  the  truck  with  your  gun  and  press  the  key  labeled  H  to 
attempt  to  identify  it  as  a  change  with  high  confidence.  You  will  not  hear  any 
confirmation  sounds,  because  the  truck  is  too  far  away.  In  a  few  minutes,  after 
we  have  started  moving  and  are  a  little  closer,  I  will  ask  you  again  to  attempt  to 
identify  the  truck.”; 

playSound  “truckTest”; 

sleep  25; 

hint  “Today’s  patrol  will  follow  the  same  route  as  your  last  session  and  as 
seen  on  the  map  at  your  training  station.  Identify  any  changes  from  the  baseline 
environment  during  this  12-minute  patrol.”; 

playSound  “patrolTime”; 

sleep  13; 

hint  “Now,  press  1  then  9  to  start  your  patrol  up  Route  Blue”; 
playSound  “oneNine”; 
while  {alive  player}  do 
{ 

_headDir  =  getDir  player; 

_vehDir  =  getDir  veh_1 ; 

_relDir  =  _headDir  -  _vehDir; 

_time  =  time; 

_append  = 

format[“AppendLine(%1)@%2,%3,”_filename2,_relDir,_time]; 

_writeFile  =  pluginFunction  [“VBSPIuginFileAccess” ,  _append]; 
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sleep  0.2; 


}; 


3.  Session  3 

0x2E  bindKey  “nul  =  []  execVM  ‘lookatl.sqf”; 

0x2 F  bindKey  “nul  =  []  execVM  ‘lookat2.sqf”; 

0x30  bindKey  “nul  =  []  execVM  ‘lookat3.sqf”; 

myArray  = 

[tire_1  ,bumt_truck_1  ,propane_tank_sales_1  ,broken_wall_1  ,dirt_pile_1  ,vid_came 
ra_1  ,observer_1  ,running_man_1  ,ied_box_1  ,ied_supply_man_1  ,odd_drums_1  ,vb 
ied_van_1  ,ied_patch_1  ,burkha_girl_1  ,ruined_house_1  ,broken_wall_2,bad_trash 
_pile_1  ,broken_wall_ied_1  ,broken_guardrail_1  ,bad_curb_1  ,reporter_1  ,bad_bag_ 
pile_1, 

bad_barrels_1  ,bad_rubble_1  ,broken_fence_no_ip_1  ,buried_tank_1  ,new_contain 
ers_1  ,poster_wall_1  ,concrete_ied_1  ,bad_mixer_1]; 

_blank  = 

total  Clicks  =  0; 

total  Correct  =  0; 

scoreL  =  0; 

scoreM  =  0; 

score  H  =  0; 

clickL  =  0; 

clickM  =  0; 

clickH  =  0; 

_filename  =  “targetsFound.txt”; 
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_filename2  =  “headDirection.txt”; 

_filename3  =  “clicked.txt”; 

_filename4  =  “startTimes.txt”; 

_append  =  format[“Write(%1)@%2,”_filename,_blank]; 

_writeFile  =  pluginFunction  [“VBSPIuginFileAccess” ,  _append]; 

_append  =  format[“Write(%1)@%2,”_filename2,_blank]; 

_writeFile  =  pluginFunction  [“VBSPIuginFileAccess” ,  _append]; 

_append  =  format[“Write(%1)@%2,”_filename3,_blank]; 

_writeFile  =  pluginFunction  [“VBSPIuginFileAccess” ,  _append]; 

_append  =  format[“Write(%1)@%2,”_filename4,_blank]; 

_writeFile  =  pluginFunction  [“VBSPIuginFileAccess”  ,  _append]; 

hint  “Welcome  to  day  three  of  Change  Detection  Training.  In  session  one, 
you  observed  the  baseline  environment.  In  your  last  session,  you  attempted  to 
identify  changes  in  the  environment.  It  was  likely  a  little  frustrating  since  the 
environment  is  still  so  new  to  you.  However,  with  each  session  you  should 
become  more  accustomed  to  the  sights  and  activity  of  the  patrol  area.  Today, 
you  will  have  another  opportunity  to  identify  changes  from  the  baseline 
environment  that  you  observed  during  the  first  session  of  this  week.”; 

playSound  “welcomeDayThree”; 

sleep  30; 

hint  “As  a  reminder,  to  identify  changes,  you  will  aim  the  gun  at  the 
location  and  then  press  the  key  labeled  H,  M,  or  L  according  to  your  confidence 
level  that  something  has  changed  from  the  baseline  session  —  High  Confidence, 
Mid-Level  Confidence,  Low  Confidence,  respectively.”; 

playSound  “conflevelsThree”; 
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sleep  19; 

hint  “Remember  from  the  instruction  sheet  you  received,  that  the  target 
you  want  to  identify  must  be  within  50  meters  of  your  location  and  within  a  15 
degree  cone  for  the  system  to  register  a  hit,  even  if  it  is  an  actual  change.  If  you 
have  identified  an  actual  change  within  the  specified  parameters,  you  will  hear  a 
short  sound  confirming  the  hit.  If  you  press  a  button  while  targeting  anything  else 
that  is  not  a  change,  or  are  outside  the  parameters,  you  will  hear  nothing.”; 

playSound  “targetingThree”; 

sleep  27; 

hint  “Today’s  patrol  will  follow  the  same  route  as  your  last  session  and  as 
seen  on  the  map  at  your  training  station.  Identify  any  changes  from  the  baseline 
environment  during  this  12-minute  patrol.  You  may  now  begin  your  patrol  up 
Route  Blue  by  pressing  1  then  9.  Good  Luck”; 

playSound  “patrolGo”; 

while  {alive  player}  do 

{ 

_headDir  =  getDir  player; 

_vehDir  =  getDir  veh_1 ; 

_relDir  =  _headDir  -  _vehDir; 

_time  =  time; 

_append  = 

format[“AppendLine(%1)@%2,%3,”_filename2,_relDir,_time]; 

_writeFile  =  pluginFunction  [“VBSPIuginFileAccess” ,  _append]; 

sleep  0.2; 

}; 
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4. 


Session  4 


_blank  = 

_filename2  =  “headDirection.txt”; 

_append  =  format[“Write(%1)@%2,”_filename2,_blank]; 

_writeFile  =  pluginFunction  [“VBSPIuginFileAccess”  ,  _append]; 

hint  “Welcome  to  the  second  week  of  Change  Detection  Training.  This  is 
your  first  session  of  three  sessions  for  this  week.  Like  last  week,  in  this  first 
session  you  will  be  exposed  to  the  baseline  environment.  It  is  exactly  the  same 
baseline  that  you  saw  in  last  week’s  first  session.  Therefore,  today  your  task  is 
simply  to  observe  the  environment  so  that  you  are  prepared  to  identify  changes 
in  the  next  two  sessions.  You  will  not  be  clicking  or  pressing  H,  M,  or  L  for 
anything  today.  When  you  are  ready  to  begin,  press  1  then  9.”; 

playSound  “weekTwoSessionOne”; 

while  {alive  player}  do 

{ 

_headDir  =  getDir  player; 

_vehDir  =  getDir  veh_1 ; 

_relDir  =  _headDir  -  _vehDir; 

_time  =  time; 

_append  = 

format[“AppendLine(%1)@%2,%3,”_filename2,_relDir,_time]; 

_writeFile  =  pluginFunction  [“VBSPIuginFileAccess” ,  _append]; 

sleep  0.2; 


5. 


Session  5 


0x2E  bindKey  “nul  =  []  execVM  ‘lookatl  .sqf”; 

0x2 F  bindKey  “nul  =  []  execVM  1ookat2.sqf”; 

0x30  bindKey  “nul  =  []  execVM  1ookat3.sqf”; 
myArray  =  []; 

_blank  = 
totalClicks  =  0; 
totalCorrect  =  0; 
scoreL  =  0; 
scoreM  =  0; 
scoreH  =  0; 
clickL  =  0; 
clickM  =  0; 
clickH  =  0; 

_filename  =  “targetsFound.txt”; 

_filename2  =  “headDirection.txt”; 

_filename3  =  “clicked.txt”; 

_filename4  =  “startTimes.txt”; 

_append  =  format[“Write(%1)@%2,”_filename,_blank]; 
_writeFile  =  pluginFunction  [“VBSPIuginFileAccess”  ,  _append]; 
_append  =  format[“Write(%1)@%2,”_filename2,_blank]; 
_writeFile  =  pluginFunction  [“VBSPIuginFileAccess”  ,  _append]; 
_append  =  format[“Write(%1)@%2,”_filename3,_blank]; 
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_writeFile  =  pluginFunction  [“VBSPIuginFileAccess”  ,  _append]; 

_append  =  format[“Write(%1)@%2,”_filename4,_blank]; 

_writeFile  =  pluginFunction  [“VBSPIuginFileAccess”  ,  _append]; 

hint  “Welcome  to  day  two  of  week  2  of  Change  Detection  Training.  In  your 
last  session,  you  were  allowed  to  observe  the  baseline  environment.  In  today’s 
session,  you  will  attempt  to  identify  any  changes  from  that  baseline  environment. 
Use  the  H,  M,  and  L  keys  to  indicate  your  confidence  level  when  identifying  a 
suspected  change.”; 

playSound  “welcomeDayTwo”; 

sleep  20; 

hint  “Now,  press  1  then  9  to  start  your  patrol  up  Route  Blue”; 
playSound  “oneNine”; 
while  {alive  player}  do 
{ 

_headDir  =  getDir  player; 

_vehDir  =  getDir  veh_1 ; 

_relDir  =  _headDir  -  _vehDir; 

_time  =  time; 

_append  = 

format[“AppendLine(%1)@%2,%3,”_filename2,_relDir,_time]; 

_writeFile  =  pluginFunction  [“VBSPIuginFileAccess”  ,  _append]; 

sleep  0.2; 

}; 
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6. 


Session  6 


0x2E  bindKey  “nul  =  []  execVM  ‘lookatl  .sqf”; 

0x2 F  bindKey  “nul  =  []  execVM  ‘lookat2.sqf”; 

0x30  bindKey  “nul  =  []  execVM  ‘lookat3.sqf”; 

myArray  = 

[fridge_no_play,truck_wreck,suicide_bomber,dead_body,bomb_maker,new_rock, 
new_van,boarded_up_shop,weird_dudes,bomb_car,press_man,back_to_back_v 
ans,UN_sacks,ice_cream_dude,bad_pile,new_boxes_no_phone_booth,new_trac 
tor,new_car_open_street,nobody_ia_cp,broken_wall,jingle_truck,turned_truck,ne 
w_rubble_no_kids, 

wall_truck,ip_suicide_bomber,propane_tanks,bad_cart,van_bomb,new_cars,new 

_barrels]; 

_blank  = 

total  Clicks  =  0; 

total  Correct  =  0; 

scoreL  =  0; 

scoreM  =  0; 

score  H  =  0; 

clickL  =  0; 

clickM  =  0; 

clickH  =  0; 

_filename  =  “targetsFound.txt”; 

_filename2  =  “headDirection.txt”; 

_filename3  =  “clicked.txt”; 

_filename4  =  “startTimes.txt”; 
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_append  =  format[“Write(%1)@%2,”_filename,_blank]; 

_writeFile  =  pluginFunction  [“VBSPIuginFileAccess” ,  _append]; 

_append  =  format[“Write(%1  )@%2,”_filename2,_blank]; 

_writeFile  =  pluginFunction  [“VBSPIuginFileAccess”  ,  _append]; 

_append  =  format[“Write(%1)@%2,”_filename3,_blank]; 

_writeFile  =  pluginFunction  [“VBSPIuginFileAccess” ,  _append]; 

_append  =  format[“Write(%1)@%2,”_filename4,_blank]; 

_writeFile  =  pluginFunction  [“VBSPIuginFileAccess”  ,  _append]; 

hint  “This  is  Session  3  and  the  last  session  for  this  week.  Like  your  last 
session,  in  today’s  session  you  will  be  asked  to  identify  changes  according  to 
your  confidence  level.  Identify  any  changes  you  see  from  the  baseline 
environment  you  observed  in  the  first  session  this  week.  Press  1  then  9  when 
you  are  ready  to  begin”; 

playSound  “welcomeDayThree”; 

sleep  22; 

while  {alive  player}  do 

{ 

_headDir  =  getDir  player; 

_vehDir  =  getDir  veh_1 ; 

_relDir  =  _headDir  -  _vehDir; 

_time  =  time; 

_append  = 

format[“AppendLine(%1)@%2,%3,”_filename2,_relDir,_time]; 

_writeFile  =  pluginFunction  [“VBSPIuginFileAccess” ,  _append]; 
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sleep  0.2; 


}; 


7.  Session  7 

_blank  = 

_filename2  =  “headDirection.txt”; 

_append  =  format[“Write(%1)@%2,”_filename2,_blank]; 

_writeFile  =  pluginFunction  [“VBSPIuginFileAccess”  ,  _append]; 

hint  “Welcome  to  the  third  week  of  Change  Detection  Training.  This  is 
your  first  session  of  three  sessions  for  this  week.  Like  last  week,  in  this  first 
session  you  will  be  exposed  to  the  baseline  environment.  It  is  exactly  the  same 
baseline  that  you  saw  in  the  last  two  weeks’  first  session.  Therefore,  today  your 
task  is  simply  to  observe  the  environment  so  that  you  are  prepared  to  identify 
changes  in  the  next  two  sessions.  You  will  not  be  clicking  or  pressing  H,  M,  or  L 
for  anything  today.  When  you  are  ready  to  begin,  press  1  then  9.”; 

playSound  “weekTwoSessionOne”; 

while  {alive  player}  do 

{ 

_headDir  =  getDir  player; 

_vehDir  =  getDir  veh_1 ; 

_relDir  =  _headDir  -  _vehDir; 

_time  =  time; 

_append  = 

format[“AppendLine(%1  )@%2,%3,”_filename2,_relDir,_time]; 

_writeFile  =  pluginFunction  [“VBSPIuginFileAccess”  ,  _append]; 
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sleep  0.2; 


}; 

8.  Session  8 

0x2E  bindKey  “nul  =  []  execVM  ‘lookatl  .sqf”; 

0x2 F  bindKey  “nul  =  []  execVM  1ookat2.sqf”; 

0x30  bindKey  “nul  =  []  execVM  1ookat3.sqf”; 
myArray  =  []; 

_blank  = 
totalClicks  =  0; 
totalCorrect  =  0; 
scoreL  =  0; 
scoreM  =  0; 
scoreH  =  0; 
clickL  =  0; 
clickM  =  0; 
clickH  =  0; 

_filename  =  “targetsFound.txt”; 

_filename2  =  “headDirection.txt”; 

_filename3  =  “clicked.txt”; 

_filename4  =  “startTimes.txt”; 

_append  =  format[“Write(%1)@%2,”_filename,_blank]; 
_writeFile  =  pluginFunction  [“VBSPIuginFileAccess”  ,  _append]; 
_append  =  format[“Write(%1)@%2,”_filename2,_blank]; 
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_writeFile  =  pluginFunction  [“VBSPIuginFileAccess”  ,  _append]; 

_append  =  format[“Write(%1)@%2,”_filename3,_blank]; 

_writeFile  =  pluginFunction  [“VBSPIuginFileAccess”  ,  _append]; 

_append  =  format[“Write(%1)@%2,”_filename4,_blank]; 

_writeFile  =  pluginFunction  [“VBSPIuginFileAccess” ,  _append]; 

hint  “Welcome  to  day  two  of  week  3  of  Change  Detection  Training.  In  your 
last  session,  you  were  allowed  to  observe  the  baseline  environment.  In  today’s 
session,  you  will  attempt  to  identify  any  changes  from  that  baseline  environment. 
Use  the  H,  M,  and  L  keys  to  indicate  your  confidence  level  when  identifying  a 
suspected  change.  Remember  that  misses  count  against  your  score  so  you 
should  have  some  level  of  confidence  that  something  has  changed  before 
pressing  a  key.”; 

playSound  “welcomeDayTwo”; 

sleep  28; 

hint  “Now,  press  1  then  9  to  start  your  patrol  up  Route  Blue”; 
playSound  “oneNine”; 
while  {alive  player}  do 
{ 

_headDir  =  getDir  player; 

_vehDir  =  getDir  veh_1 ; 

_relDir  =  _headDir  -  _vehDir; 

_time  =  time; 

_append  = 

format[“AppendLine(%1)@%2,%3,”_filename2,_relDir,_time]; 

_writeFile  =  pluginFunction  [“VBSPIuginFileAccess”  ,  _append]; 
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sleep  0.2; 


}; 


9.  Session  9 

0x2E  bindKey  “nul  =  []  execVM  ‘lookatl.sqf”; 

0x2 F  bindKey  “nul  =  []  execVM  ‘lookat2.sqf”; 

0x30  bindKey  “nul  =  []  execVM  ‘lookat3.sqf”; 

myArray  = 

[tire_1  ,bumt_truck_1  ,propane_tank_sales_1  ,broken_wall_1  ,dirt_pile_1  ,vid_came 
ra_1  ,observer_1  ,running_man_1  ,ied_box_1  ,ied_supply_man_1  ,odd_drums_1  ,vb 
ied_van_1  ,ied_patch_1  ,burkha_girl_1  ,ruined_house_1  ,broken_wall_2,bad_trash 
_pile_1  ,broken_wall_ied_1  ,broken_guardrail_1  ,bad_curb_1  ,reporter_1  ,bad_bag_ 
pile_1, 

bad_barrels_1  ,bad_rubble_1  ,broken_fence_no_ip_1  ,buried_tank_1  ,new_contain 
ers_1  ,poster_wall_1  ,concrete_ied_1  ,bad_mixer_1]; 

_blank  = 

total  Clicks  =  0; 

total  Correct  =  0; 

scoreL  =  0; 

scoreM  =  0; 

score  H  =  0; 

clickL  =  0; 

clickM  =  0; 

clickH  =  0; 

_filename  =  “targetsFound.txt”; 
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_filename2  =  “headDirection.txt”; 

_filename3  =  “clicked.txt”; 

_filename4  =  “startTimes.txt”; 

_append  =  format[“Write(%1)@%2,”_filename,_blank]; 

_writeFile  =  pluginFunction  [“VBSPIuginFileAccess” ,  _append]; 

_append  =  format[“Write(%1)@%2,”_filename2,_blank]; 

_writeFile  =  pluginFunction  [“VBSPIuginFileAccess” ,  _append]; 

_append  =  format[“Write(%1)@%2,”_filename3,_blank]; 

_writeFile  =  pluginFunction  [“VBSPIuginFileAccess” ,  _append]; 

_append  =  format[“Write(%1)@%2,”_filename4,_blank]; 

_writeFile  =  pluginFunction  [“VBSPIuginFileAccess” ,  _append]; 

hint  “This  is  Session  3  of  the  3rd  week  for  your  training.  Today  you  will 
again  identify  changes  according  to  your  confidence  level  that  something  has 
changed.  Remember  that  misses  count  against  your  score  so  you  should  have 
some  level  of  confidence  that  something  has  changed  before  pressing  a  key. 
Press  1  then  9  to  begin  the  patrol.”; 

playSound  “welcomeDayThree”; 

while  {alive  player}  do 

{ 

_headDir  =  getDir  player; 

_vehDir  =  getDir  veh_1 ; 

_relDir  =  _headDir  -  _vehDir; 

_time  =  time; 
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_append  = 

format[“AppendLine(%1)@%2,%3,”_filename2,_relDir,_time]; 

_writeFile  =  pluginFunction  [“VBSPIuginFileAccess” ,  _append]; 

sleep  0.2; 

}; 

10.  Session  10 

_blank  = 

_filename2  =  “headDirection.txt”; 

_append  =  format[“Write(%1)@%2,”_filename2,_blank]; 

_writeFile  =  pluginFunction  [“VBSPIuginFileAccess”  ,  _append]; 

hint  “Welcome  to  your  last  week  of  Change  Detection  Training.  This  is 
your  first  session  of  three  sessions  for  this  week.  Like  last  week,  in  this  first 
session  you  will  be  exposed  to  the  baseline  environment.  It  is  exactly  the  same 
baseline  that  you  saw  in  the  last  three  weeks’  first  session.  Therefore,  today 
your  task  is  simply  to  observe  the  environment  so  that  you  are  prepared  to 
identify  changes  in  the  next  two  sessions.  You  will  not  be  clicking  or  pressing  H, 
M,  or  L  for  anything  today.  When  you  are  ready  to  begin,  press  1  then  9.”; 

playSound  “weekTwoSessionOne”; 

while  {alive  player}  do 

{ 

_headDir  =  getDir  player; 

_vehDir  =  getDir  veh_1 ; 

_relDir  =  _headDir  -  _vehDir; 

_time  =  time; 
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_append 

format[“AppendLine(%1)@%2,%3,”_filename2,_relDir,_time]; 

_writeFile  =  pluginFunction  [“VBSPIuginFileAccess” ,  _append]; 
sleep  0.2; 

}; 

11.  Session  11 

0x2E  bindKey  “nul  =  []  execVM  ‘lookatl  .sqf”; 

0x2 F  bindKey  “nul  =  []  execVM  1ookat2.sqf”; 

0x30  bindKey  “nul  =  []  execVM  1ookat3.sqf”; 
myArray  =  []; 

_blank  = 
total  Clicks  =  0; 
total  Correct  =  0; 
scoreL  =  0; 
scoreM  =  0; 
score  H  =  0; 
clickL  =  0; 
clickM  =  0; 
clickH  =  0; 

_filename  =  “targetsFound.txt”; 

_filename2  =  “headDirection.txt”; 

_filename3  =  “clicked.txt”; 

_filename4  =  “startTimes.txt”; 
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_append  =  format[“Write(%1)@%2,”_filename,_blank]; 

_writeFile  =  pluginFunction  [“VBSPIuginFileAccess” ,  _append]; 

_append  =  format[“Write(%1)@%2,”_filename2,_blank]; 

_writeFile  =  pluginFunction  [“VBSPIuginFileAccess”  ,  _append]; 

_append  =  format[“Write(%1)@%2,”_filename3,_blank]; 

_writeFile  =  pluginFunction  [“VBSPIuginFileAccess”  ,  _append]; 

_append  =  format[“Write(%1)@%2,”_filename4,_blank]; 

_writeFile  =  pluginFunction  [“VBSPIuginFileAccess” ,  _append]; 

hint  “Welcome  to  day  two  of  the  last  week  of  Change  Detection  Training. 
In  your  last  session,  you  were  allowed  to  observe  the  baseline  environment.  In 
today’s  session,  you  will  attempt  to  identify  any  changes  from  that  baseline 
environment.  Use  the  H,  M,  and  L  keys  to  indicate  your  confidence  level  when 
identifying  a  suspected  change.  Remember  that  misses  count  against  your  score 
so  you  should  have  some  level  of  confidence  that  something  has  changed  before 
pressing  a  key.”; 

playSound  “welcomeDayTwo”; 

sleep  28; 

hint  “Now,  press  1  then  9  to  start  your  patrol  up  Route  Blue”; 
playSound  “oneNine”; 
while  {alive  player}  do 
{ 

_headDir  =  getDir  player; 

_vehDir  =  getDir  veh_1 ; 

_relDir  =  _headDir  -  _vehDir; 

_time  =  time; 
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_append 

format[“AppendLine(%1)@%2,%3,”_filename2,_relDir,_time]; 

_writeFile  =  pluginFunction  [“VBSPIuginFileAccess” ,  _append]; 
sleep  0.2; 


12.  Session  12 

0x2E  bindKey  “nul  =  []  execVM  ‘lookatl.sqf”; 

0x2 F  bindKey  “nul  =  []  execVM  1ookat2.sqf”; 

0x30  bindKey  “nul  =  []  execVM  1ookat3.sqf”; 

0x15  bindKey  “nul  =  []  execVM  ‘yes.sqf”; 

0x16  bindKey  “nul  =  []  execVM  ‘no.sqf”; 

myArray  = 

[fridge_no_play,truck_wreck,suicide_bomber,dead_body,bomb_maker,new_rock, 
new_van,boarded_up_shop,weird_dudes,bomb_car,press_man,back_to_back_v 
ans,UN_sacks,ice_cream_dude,bad_pile,new_boxes_no_phone_booth,new_trac 
tor,new_car_open_street,nobody_ia_cp,broken_wall,jingle_truck,turned_truck,ne 
w_rubble_no_kids, 

wall_truck,ip_suicide_bomber,propane_tanks,bad_cart,van_bomb,new_cars,new 

_barrels]; 

_blank  = 

total  Clicks  =  0; 

total  Correct  =  0; 

scoreL  =  0; 

scoreM  =  0; 
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score  H  =  0; 


clickL  =  0; 
clickM  =  0; 
clickH  =  0; 

_filename  =  “targetsFound.txt”; 

_filename2  =  “headDirection.txt”; 

_filename3  =  “clicked.txt”; 

_filename4  =  “startTimes.txt”; 

_filename5  =  “survey.txt”; 

_append  =  format[“Write(%1)@%2, ’’-filename, _blank]; 

-WriteFile  =  pluginFunction  [“VBSPIuginFileAccess”  ,  _append]; 

_append  =  format[“Write(%1)@%2,”_filename2,_blank]; 

-WriteFile  =  pluginFunction  [“VBSPIuginFileAccess” ,  _append]; 

_append  =  format[“Write(%1)@%2,”_filename3,_blank]; 

-WriteFile  =  pluginFunction  [“VBSPIuginFileAccess” ,  _append]; 

_append  =  format[“Write(%1)@%2,”_filename4,_blank]; 

-WriteFile  =  pluginFunction  [“VBSPIuginFileAccess”  ,  _append]; 

_append  =  format[“Write(%1)@%2,”_filename5,_blank]; 

-WriteFile  =  pluginFunction  [“VBSPIuginFileAccess”  ,  _append]; 

hint  “This  is  Session  3  and  the  last  session  for  this  week.  Like  your  last 
session,  in  today’s  session  you  will  be  asked  to  identify  changes  according  to 
your  confidence  level.  Identify  any  changes  you  see  from  the  baseline 
environment  you  observed  in  the  first  session  this  week.  Press  1  then  9  when 
you  are  ready  to  begin”; 
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playSound  “welcomeDayThree”; 
sleep  22; 

while  {alive  player}  do 

{ 

_headDir  =  getDir  player; 

_vehDir  =  getDir  veh_1 ; 

_relDir  =  _headDir  -  _vehDir; 

_time  =  time; 

_append 

format[“AppendLine(%1)@%2,%3,”_filename2,_relDir,_time]; 

_writeFile  =  pluginFunction  [“VBSPIuginFileAccess”  ,  _append]; 
sleep  0.2; 

}; 
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APPENDIX  G.  VBS2™  VOICEOVER  SCRIPTS 


A.  SCENARIO  SCRIPTS 

1.  Session  1 

“Aim  at  the  Yellow  Cross.  This  is  the  maximum  distance  at  which  the 
system  will  allow  you  to  identify  changes.  The  Green  Arrows  represent  the 
maximum  allowable  azimuth  error.” 

“In  this  scenario  today,  you  will  not  be  identifying  changes.  You  are  to 
conduct  observation  as  the  gunner  during  the  patrol.  Move  your  mouse  to  rotate 
your  view.  Learn  this  environment.  In  the  next  two  sessions,  you  will  be  required 
to  identify  things  that  have  changed  from  the  environment  you  see  today.” 

“Every  session’s  patrol  route  will  be  the  same.  It  will  last  approximately  12 
minutes.  We  will  follow  Route  Blue,  straight  ahead  of  you.  Then,  we  will  make  a 
right  turn  on  Route  Red.  Then,  when  we  reach  Route  Green  at  Victory  Bridge, 
we  will  take  another  right  and  follow  Route  Green  along  the  river.  At  the  end  of 
Route  Green,  we  will  make  a  right  turn  onto  Route  Purple  for  the  final  portion  of 
our  patrol.  Again,  every  session’s  patrol  will  follow  this  same  route.” 

“Again,  it  is  important  to  understand  that  today’s  session  exposes  you  to 
the  base  or  baseline  environment.  You  will  be  required  in  the  other  two  sessions 
of  this  week  to  identify  changes  from  this  environment  you  see  today.  As  we 
proceed  with  the  training,  you  will  see  this  baseline  environment  as  the  first 
session  of  every  week.” 

“Now,  press  1  then  9  to  start  your  patrol  up  Route  Blue” 
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2. 


Session  2 


“Welcome  to  day  two  of  Change  Detection  Training.  In  your  last  session, 
you  were  allowed  to  observe  the  baseline  environment.  In  today’s  session,  you 
will  attempt  to  identify  any  changes  from  that  baseline  environment.  Please  be 
patient  and  listen  closely  as  I  share  some  important  instructions.” 

“To  identify  changes,  you  will  aim  the  gun  at  the  location  and  then  press 
the  key  labeled  H,  M,  or  L  according  to  your  confidence  level  that  something  has 
changed  from  the  baseline  session  --  High  Confidence,  Mid-Level  Confidence, 
Low  Confidence,  respectively.” 

“Remember  from  the  instruction  sheet  you  received,  that  the  target  you 
want  to  identify  must  be  within  50  meters  of  your  location  and  within  a  15  degree 
cone  for  the  system  to  register  a  hit,  even  if  it  is  an  actual  change.  Displayed  right 
now  for  your  understanding  are  green  arrows  ahead  of  you  on  the  road 
displaying  the  limits  of  this  area.  If  you  try  to  identify  a  target  outside  these 
parameters,  it  will  not  register  as  a  hit  and  will  in  fact  register  as  a  miss.  Misses 
will  count  against  your  total  score.  It  is  best  to  wait  until  you  have  the  location  you 
want  to  target  reasonably  close  before  pressing  the  desired  key.” 

“I  will  now  remove  the  arrows” 

“If  you  have  identified  an  actual  change  within  the  specified  parameters, 
you  will  hear  a  short  ding  confirming  the  hit.  If  you  press  a  button  while  targeting 
anything  else,  or  are  outside  the  parameters,  you  will  hear  nothing.  Do  not  get 
frustrated,  you  have  only  seen  the  baseline  once  so  far  in  this  first  week  of 
training.  Just  continue  to  learn  the  baseline  environment  better  each  time  you 
see  it  as  the  first  session  of  a  week.” 

“There  is  a  white  pick-up  truck  to  the  right  of  the  road  ahead  of  you.  Go 
ahead  and  aim  at  the  truck  with  your  gun  and  press  the  key  labeled  H  to  attempt 
to  identify  it  as  a  change  with  high  confidence.  You  will  not  hear  any  confirmation 
sounds,  because  the  truck  is  too  far  away.  In  a  few  minutes,  after  we  have 
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started  moving  and  are  a  little  closer,  I  will  ask  you  again  to  attempt  to  identify  the 
truck.” 

“Today’s  patrol  will  follow  the  same  route  as  your  last  session  and  as  seen 
on  the  map  at  your  training  station.  Identify  any  changes  from  the  baseline 
environment  during  this  12-minute  patrol.” 

“Now,  press  1  then  9  to  start  your  patrol  up  Route  Blue” 

“Ok,  you  should  be  close  enough  to  identify  the  truck.  We  are  highly 
confident  of  the  change,  so  press  the  key  labeled  “H”  while  aiming  at  the  truck  to 
indicate  you  are  highly  confident  that  the  truck  is  a  change” 

“You  should  hear  a  confirmation  “ding”  which  indicates  that  was  a  correct 
detection” 

“You  will  now  conduct  the  patrol.  Identify  any  changes  you  may  see. 
Good  luck.” 

3.  Session  3 

“Welcome  to  day  three  of  Change  Detection  Training.  In  session  one,  you 
observed  the  baseline  environment.  In  your  last  session,  you  attempted  to 
identify  changes  in  the  environment.  It  was  likely  a  little  frustrating  since  the 
environment  is  still  so  new  to  you.  However,  with  each  session  you  should 
become  more  accustomed  to  the  sights  and  activity  of  the  patrol  area.  Today, 
you  will  have  another  opportunity  to  identify  changes  from  the  baseline 
environment  that  you  observed  during  the  first  session  of  this  week.” 

“As  a  reminder,  to  identify  changes,  you  will  aim  the  gun  at  the  location 
and  then  press  the  key  labeled  H,  M,  or  L  according  to  your  confidence  level  that 
something  has  changed  from  the  baseline  session  -  High  Confidence,  Mid-Level 
Confidence,  Low  Confidence,  respectively.” 

“Remember  from  the  instruction  sheet  you  received,  that  the  target  you 

want  to  identify  must  be  within  50  meters  of  your  location  and  within  a  15  degree 
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cone  for  the  system  to  register  a  hit,  even  if  it  is  an  actual  change.  If  you  have 
identified  an  actual  change  within  the  specified  parameters,  you  will  hear  a  short 
sound  confirming  the  hit.  If  you  press  a  button  while  targeting  anything  else  that 
is  not  a  change,  or  are  outside  the  parameters,  you  will  hear  nothing.” 

“Today’s  patrol  will  follow  the  same  route  as  your  last  session  and  as  seen 
on  the  map  at  your  training  station.  Identify  any  changes  from  the  baseline 
environment  during  this  12-minute  patrol.  You  may  now  begin  your  patrol  up 
Route  Blue  by  pressing  1  then  9.  Good  Luck” 

4.  Session  4 

“Welcome  to  the  second  week  of  Change  Detection  Training.  This  is  your 
first  session  of  three  sessions  for  this  week.  Like  last  week,  in  this  first  session 
you  will  be  exposed  to  the  baseline  environment.  It  is  exactly  the  same  baseline 
that  you  saw  in  last  week’s  first  session.  Therefore,  today  your  task  is  simply  to 
observe  the  environment  so  that  you  are  prepared  to  identify  changes  in  the  next 
two  sessions.  You  will  not  be  clicking  or  pressing  H,  M,  or  L  for  anything  today. 
When  you  are  ready  to  begin,  press  1  then  9.” 

5.  Session  5 

“Welcome  to  day  two  of  week  2  of  Change  Detection  Training.  In  your  last 
session,  you  were  allowed  to  observe  the  baseline  environment.  In  today’s 
session,  you  will  attempt  to  identify  any  changes  from  that  baseline  environment. 
Use  the  H,  M,  and  L  keys  to  indicate  your  confidence  level  when  identifying  a 
suspected  change.” 

“Now,  press  1  then  9  to  start  your  patrol  up  Route  Blue” 

6.  Session  6 

“This  is  Session  3  and  the  last  session  for  this  week.  Like  your  last 
session,  in  today’s  session  you  will  be  asked  to  identify  changes  according  to 
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your  confidence  level.  Identify  any  changes  you  see  from  the  baseline 
environment  you  observed  in  the  first  session  this  week.  Press  1  then  9  when 
you  are  ready  to  begin” 

7.  Session  7 

“Welcome  to  the  third  week  of  Change  Detection  Training.  This  is  your 
first  session  of  three  sessions  for  this  week.  Like  last  week,  in  this  first  session 
you  will  be  exposed  to  the  baseline  environment.  It  is  exactly  the  same  baseline 
that  you  saw  in  the  last  two  weeks’  first  session.  Therefore,  today  your  task  is 
simply  to  observe  the  environment  so  that  you  are  prepared  to  identify  changes 
in  the  next  two  sessions.  You  will  not  be  clicking  or  pressing  H,  M,  or  L  for 
anything  today.  When  you  are  ready  to  begin,  press  1  then  9.” 

8.  Session  8 

“Welcome  to  day  two  of  week  3  of  Change  Detection  Training.  In  your  last 
session,  you  were  allowed  to  observe  the  baseline  environment.  In  today’s 
session,  you  will  attempt  to  identify  any  changes  from  that  baseline  environment. 
Use  the  H,  M,  and  L  keys  to  indicate  your  confidence  level  when  identifying  a 
suspected  change.  Remember  that  misses  count  against  your  score  so  you 
should  have  some  level  of  confidence  that  something  has  changed  before 
pressing  a  key.” 

“Now,  press  1  then  9  to  start  your  patrol  up  Route  Blue” 

9.  Session  9 

“This  is  Session  3  of  the  3rd  week  for  your  training.  Today  you  will  again 
identify  changes  according  to  your  confidence  level  that  something  has  changed. 
Remember  that  misses  count  against  your  score  so  you  should  have  some  level 
of  confidence  that  something  has  changed  before  pressing  a  key.  Press  1  then  9 
to  begin  the  patrol.” 
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10.  Session  10 


“Welcome  to  your  last  week  of  Change  Detection  Training.  This  is  your 
first  session  of  three  sessions  for  this  week.  Like  last  week,  in  this  first  session 
you  will  be  exposed  to  the  baseline  environment.  It  is  exactly  the  same  baseline 
that  you  saw  in  the  last  three  weeks’  first  session.  Therefore,  today  your  task  is 
simply  to  observe  the  environment  so  that  you  are  prepared  to  identify  changes 
in  the  next  two  sessions.  You  will  not  be  clicking  or  pressing  H,  M,  or  L  for 
anything  today.  When  you  are  ready  to  begin,  press  1  then  9.” 

11.  Session  11 

“Welcome  to  day  two  of  the  last  week  of  Change  Detection  Training.  In 
your  last  session,  you  were  allowed  to  observe  the  baseline  environment.  In 
today’s  session,  you  will  attempt  to  identify  any  changes  from  that  baseline 
environment.  Use  the  H,  M,  and  L  keys  to  indicate  your  confidence  level  when 
identifying  a  suspected  change.  Remember  that  misses  count  against  your  score 
so  you  should  have  some  level  of  confidence  that  something  has  changed  before 
pressing  a  key.” 

“Now,  press  1  then  9  to  start  your  patrol  up  Route  Blue” 

12.  Session  12 

“This  is  Session  3  and  the  last  session  for  this  week.  Like  your  last 
session,  in  today’s  session  you  will  be  asked  to  identify  changes  according  to 
your  confidence  level.  Identify  any  changes  you  see  from  the  baseline 
environment  you  observed  in  the  first  session  this  week.  Press  1  then  9  when 
you  are  ready  to  begin” 
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B.  TUTORIAL  SCRIPTS 


1.  Week  1  Baseline  Tutorial 

“Welcome  to  a  short  tutorial  for  Change  Detection  training” 

“Recognizing  change  in  a  complex  environment  can  be  difficult  but  it  is  an 
important  skill  for  soldiers  or  marines  operating  in  the  same  dangerous 
neighborhoods  day  after  day.  Some  changes  may  be  indicators  of  a  threat, 
some  changes  may  be  significant  to  report  as  measures  of  operational 
effectiveness  in  an  area” 

“In  this  tutorial,  you  will  first  be  exposed  to  a  baseline  situation.  I  will  give 
you  pointers  and  hints  on  what  to  observe.  Then,  we  will  load  a  new  scenario 
with  changes  which  I  will  help  you  identify.” 

“First,  note  the  two  men  to  your  right  and  what  they  are  wearing” 

“Look  at  the  vehicle  to  your  left  and  note  the  color  and  type” 

“Note  the  build-up  of  something  at  the  edge  of  the  building  to  the  left” 

“Note  the  behavior  of  the  man  with  the  vest  ahead  of  you  in  the  road” 

“Note  the  fridge  standing  to  your  right” 

“Note  the  three  plants  on  the  wall  to  the  left” 

“Note  the  donkey  and  the  cart  in  the  garage  to  the  right” 

“Note  the  nature  of  the  box  in  a  strange  spot  on  your  left  and  the  people 
standing  near  it” 

“Now  let’s  close  this  scenario  and  load  one  up  with  changes  and  see  what 
we  can  identify” 

2.  Week  1  Change  Tutorial 

“Ok,  now  things  have  changed.” 
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“Remember  the  two  men  on  the  right?  That  has  changed,  right?  Now 
there  is  some  important  looking  person  in  the  neighborhood” 

“Look  at  the  vehicle  to  your  left  and  notice  it  has  changed.  Could  a  large 
truck  like  this  indicate  something?” 

“See  the  new  fuel  can  at  the  edge  of  the  building  to  the  left?  That  could 
be  an  indicator  of  an  immediate  threat.” 

“Notice  that  the  man  with  the  vest  is  now  displaying  different  behavior? 
Dramatic  changes  in  a  person’s  behavior  may  indicate  a  more  important  but  less 
visible  change.” 

“Note  the  fridge  is  busted  on  your  right.  If  someone  tampered  with  it,  it 
could  be  dangerous” 

“Note  the  three  plants  on  the  wall  to  the  left  are  gone  and  different  objects 
are  there.  Sometimes  small  changes  warrant  closer  inspection.” 

“Look,  the  donkey  is  dead  and  new  objects  are  in  the  cart.  Sometimes 
there  are  multiple  associated  indicators  of  a  problem” 

“Note  the  box  is  now  a  possible  immediate  threat.  Another  key  change 
here  is  that  the  people  there  previously  are  not  around  anymore.  They  probably 
know  to  stay  away!  Many  times,  when  something  or  someone  is  missing  from 
the  scene,  you  need  to  look  for  other  changes” 

“This  concludes  the  tutorial.  You  should  have  a  better  understanding  of 
what  kind  of  changes  to  look  for.  Memory  is  a  function  of  attention.  The  most 
important  thing  you  can  do  to  ensure  you  will  recognize  change  is  to  mentally 
note  a  number  of  key  locations  in  a  neighborhood  while  on  patrol.  As  you  spend 
more  time  in  the  neighborhood,  you  can  increase  your  awareness  of  what  normal 
may  look  like.  You  will  then  be  better  at  identifying  changes  which  may  indicate 
threats  or  other  significant  events.” 
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3. 


Week  3  Baseline  Tutorial 


“This  is  the  second  tutorial  session  for  change  detection  training.  You 
have  some  experience  now,  at  least  within  a  virtual  environment  identifying 
change.  For  this  brief  tutorial  I  will  again  show  you  some  ways  to  identify 
changes  in  your  environment.  You  have  likely  discovered  that  memory  is  critical 
to  identifying  changes.  You  have  also  learned  that  memory  is  a  function  of 
attention.  But,  attention  is  a  limited  resource.  So  how  should  we  best  direct  our 
attention  when  observing  our  environment?” 

“As  you  have  discovered,  even  this  virtual  depiction  of  a  typical  Middle 
Eastern  neighborhood  is  cluttered  with  endless  things  to  pay  attention  to. 
However,  one  of  the  best  indicators  that  something  is  amiss,  or  that  something  is 
different,  is  the  local  population.  They  definitely  know  when  something  has 
changed  around  them,  especially  if  it  relates  to  some  kind  of  threat.  Other  than 
the  local  population  and  their  behavior,  you  should  pay  attention  to  significant 
infrastructure,  where  the  local  law  enforcement  are  positioning  themselves,  or 
even  change  in  activity  in  commercial  areas.” 

“We  see  we  are  starting  in  a  residential  area.  Notice  a  few  people  in  the 
homes  around  but  that  it  mostly  looks  vacant.” 

“Look  to  the  left  and  the  electrical  station.  There  doesn’t  seem  to  be  any 
activity.  You  should  remember  that,  this  may  be  affecting  power  supply  in  the 
area.” 

“Look  to  the  area  to  your  right.  It  would  be  almost  impossible  to  remember 
the  position,  type,  and  color  of  all  that  clutter.  However,  note  the  two  kids 
hanging  out.  That  should  be  easy  to  commit  to  memory.” 

“Look  ahead  at  the  traffic  pattern  of  the  vehicles.  They  obviously  feel 
comfortable  using  that  route.” 
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“Look  at  the  vehicles  and  general  activity  of  the  shipping  area  to  your  left. 
Just  take  note  that  there  is  some,  maybe  just  a  little,  but  some  commercial 
activity.” 

“We  will  now  exit  this  baseline  scenario  and  load  the  changed  scenario.” 

4.  Week  3  Change  Tutorial 

“Now,  we  will  look  at  some  changes.” 

“You  can  see  that  there  is  now  more  activity  in  the  neighborhood.  In  terms 
of  what  we  could  clearly  identify  as  representative  of  this  change  with  some 
confidence,  is  the  party  going  on  in  the  house  to  the  left.  This  increase  in 
neighborhood  activity  could  be  a  positive  indicator  that  security  is  improved.” 

“Now  we  see  what  looks  like  repair  activity  going  on  at  the  power  station. 
This  may  mean  that  security  has  improved  enough  for  the  city  to  start 
reestablishing  infrastructure.  This  is  an  important  measure  of  our  operational 
effectiveness  in  the  area.” 

“Look  to  your  right.  We  chose  to  remember  the  two  kids.  Well,  they  aren’t 
there  anymore.  If  we  use  this  as  a  clue,  we  can  look  a  little  closer  and  see  the 
disturbed  earth  near  the  refrigerator  which  may  indicate  a  threat.  This  might  be 
why  the  kids  are  not  hanging  out  any  more.” 

“Look  ahead  at  the  traffic  pattern  of  the  vehicles.  They  obviously  no 
longer  feel  comfortable  using  that  route.  If  we  use  this  as  an  indicator,  we  might 
see  the  concrete  block  near  the  trash  pile  on  the  left  which  could  be  a  threat.” 

“There  seems  to  be  increased  activity  in  the  shipping  yard.  We  could 
identify  the  tractor  truck  as  a  significant  change.” 

“This  ends  the  last  tutorial.  Since  attention  is  a  limited  resource, 
remember  to  use  it  wisely.  Rather  than  attempt  to  remember  every  specific 
thing,  think  more  about  the  patterns  of  life  in  the  area  and  what  those  changes 
may  mean.” 
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APPENDIX  H.  JMP  RAW  DATA  BY  PARTICIPANT 


A.  EXPERIMENTAL  DATA 


® 

Age 

Gender 

Service 

Branch 

Rank 

Computer 
Video  Games 

FPS 

Games 

Gamer 

? 
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Computer  Based 
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Group 

1 

37 
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04 

No 
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No 
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No  Tram  mg 

2 

38 

Male 

Army 

04 
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Weeks 
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Strongly  Agree 

Tram  mg 

3 

35 

Male 

Marine 

04 

Yes 
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No 

Yes 
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No  Trammg 

4 

29 

Male 

Navy 

03 

No 

N/A 

Yes 

No 

Strongly  Dsagree 

No  Trammg 

5 

34 

Male 

Intematxxi 

04 

Yes 
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No 

No 

Moderately  Ag-ee 

Trammg 

e 

39 

Male 

Navy 

04 

No 

N/A 

No 

No 

No  Opmton 

Trammg 

7 

41 

Male 

Army 

04 

Yes 

Years 

Yes 

No 

Strongly  Agree 

No  Trammg 

8 

28 

Male 

A*-  Force 

03 

No 

Years 

No 

No 

Moderately  Ag-ee 

No  Trammg 

9 

30 

Male 

Army 

03 

No 

N/A 

No 

Yes 

Moderately  Agree 

Trammg 

10 

32 

Male 

Navy 

03 

No 

Hours 

No 

No 

Moderately  Agree 

Trammg 

11 

39 

Male 

A*-  Force 

04 

Yes 

Hours 

No 

No 

Moderately  Agree 

Trammg 

12 

39 

Male 

Marine 

04 

Yes 

Weeks 

Yes 

No 

Strongly  Agree 

No  Trammg 

13 

28 

Male 

Intemation 

02 

Yes 

Months 

No 

No 

Moderately  Ag'ee 

No  Trammg 

14 

29 

Male 

Navy 

03 

Yes 

Years 

Yes 

No 

Moderately  Agree 

Trammg 

15 

32 

Male 

Navy 

03 

Yes 

Days 

Yes 

No 

Moderately  Agree 

No  Trammg 

Figure  32.  JMP  raw  data:  participant  demographics 
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:5:02:45 

23:15 

22:00 

:3:03:00 

48:05 

23:40 

25:56 

3 

23:00 

23:15 

21:38 

23:53 

20:30 

25:30 

:5:02:15  21:00 

25:30 

:5:01:15 

19:20 

23:55 

:4:00:45 

24:40 

23:05 

22:56 

4 
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24:00 
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0 
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24:10 

23:41 

9 

24:00 

24:00 

24:00 

22:13 

24:00 

24:00 

:5:00:00  24:00 

24:00 

:6:00:00 

24:00 

24:00 

:3:23:50 

24:10 

20:15 

23:33 

10 

36:00 

24:03 

23:35 

23:45 

48:00 

24:00 

:3:23:45  24:15 

23:50 

:5:00:55 

23:10 

24:00 

:5:00:05 

24:00 

23:30 

26:51 

11 

33:53 

23:45 

23:45 

25:38 

43:45 

24:00 

:5:04:15  24:00 

23:30 

:3:23:45 

22:40 

24:50 

.0:20:00 

27:30 

23:45 

26:45 

12 

24:05 

24:00 

22:13 

23:50 

24:30 

23:40 

:6:00:05  23:45 

24:15 

:4:21:05 

20:40 

23:45 

:4:03:15 

23:45 

23:55 

23:32 

13 

24:00 

24:00 

23:08 

23:38 

24:10 

23:50 

:5:00:00  24:00 

24:00 

4:01:50 

22:55 

23:20 

:6:01:25 

23:25 

23:50 

23:41 

14 

11:50 

20:55 

33:30 

13:30 

19:10 

4:30 

:6:00:15  23:00 

18:50 

:4:05:25 

43:15 

23:45 

:3:23:15 

3:10 

23:50 

19:56 

15 

22:15 

22:38 

24:15 

25:25 

19:45 

24:45 

:5:02:30  20:45 

24:30 

:3:23:30 

27:45 
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25:15 
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Figure  33.  JMP  raw  data:  participant  times  between  sessions 


147 


Session  1:  DTG 

1  I  201 1-05-04T1 0:00 

2  2011-05-04T14:00 

3  201 1-05-04T15:00 

4  2011-05-10T12:00 

5  2011-05-10T13:45 
8  2011-05-10T14:00 

7  2011-05-10T14:45 

8  2011-05-11113:30 
3  2011-05-16T13:00 

10  201 1-05-1 7T0845 

11  2011-05-23114:45 

12  2011-05-23T15:00 

13  201 1-05-24T1 1:15 

14  2011-05-24T15:05 

15  2011-05-24T16:30 


Session  2:  DTG 

:: 

2011-05-05T13:30 
201 1-05-05T1 1 :30 
2011-05-12T1000 
201 1-05-1 1T13:45 
201 1-05-1 1T1 4: 15 
2011-05-11T1430 
2011-05-12T13:30 
201 1-05-17T13:00 
2011-05-19T0845 
201 1-05-25110:30 
201 1-05-24T1 5:30 
201 1-05-25T1 1:25 
2011-05-25T1015 
2011-05-25T12:15 


Session  3:  DTG 
201 1-05-06T1 1:00 
2011-05-08T11:15 
201 1-05-06T1 3:00 
201 1-05-1 3T09  50 
2011-05-12T13:45 
2011-05-13T10:15 
2011-05-12T14:15 
201 1-05-1 3T13:30 
2011-05-18T13:00 
201 1-05-20T08:45 
201 1-05-28T10:30 
2011-05-25T15:10 
201 1-05-28T1 1:15 
2011-05-25T14:45 
2011-05-28T13:00 


Session  4:  DTG 
2011-05-10T13:00 
201 1-05-1 1T1 4:00 
201 1-05-1 1T15: 15 
201 1-05-17T1 1:45 
2011-05-17T13:45 
2011 -05-1 7T1 4:00 
2011-05-17T14:30 
2011-05-18T13:30 
201 1-05-23T1 3:00 
201 1-05-24T08:30 
2011-05-31T14:45 
2011-05-31T15:15 
201 1-05-31T1 1:15 
2011-05-31T15:00 
2011-05-31T15:30 


Session  5:  DTG 
201 1-05-1 1T10:0 
2011-05-12T13:1 
201 1-05-12T12:1 
2011-05-13T12:0 
2011-05-18T13:4 
2011-05-18T143 
2011-05-18T14  4 
2011-05-19T12:3 
2011-05-24T13:0 
2011-05-25T084 
2011-0601T144 
2011-06-01T15:0 
2011-06-01T11:1 
2011-06-01T14:0 
2011-06-01T12:1 


Session  S:  DTG 
2011-05-12T10:15 
201 1-05-13T1 1:15 
2011-05-13T13:45 
2011-05-20T12:15 
2011-05-19T13:45 
2011-05-20T10:00 
2011-05-19T14:30 
2011-05-20T13:30 
2011-05-25T13:00 
2011-05-28T08:35 
2011-06-02T14:15 
2011-06-02T15:15 
2011-06-02711:15 
2011-06-02708:50 
2011-06-02T12:45 


Session  7:  DTG 
201 1-05-17T10:00 
2011-05-18T14:00 
201 1-05-18T15:00 
201 1-05-24T1 1:45 
201 1-05-24T1 3:45 
2011-05-24T14:30 
2011-05-24T1410 
201 1-05-24T1 3:30 
2011-05-31T13:00 
2011-05-31T09  30 
2011-06-06T1400 
201 1-06-07T1 2:20 
201 1-06-Q6T1 3:05 
2011-06-06T1415 
2011-06-06712:15 


Session  8:  DTG 
2011-05-18T1000 
2011-05-19T13.15 
201 1-05-19T10:20 
2011-05-25T09  45 
201 1-05-25T1 3:45 
2011-05-25T16:35 
201 1-05-25T1 3:00 
201 1-05-25T1 3:30 
201 1-08-01T1 3:00 
201 1-06-0 1T08: 40 
201 1-06-0 7T1 2: 40 
2011-06-08T09  00 
201 1-08-0 7T1200 
201 1-0608T09:30 
201 1-06-07T16:00 


Session  9:  DTG 
2011-05-19T1000 
201 1-05-20T1 1:15 
2011-05-20T1015 
201 1-05-26T1 1:45 
2011-05-26T13:45 
201 1-05-28T1800 
2011-05-26T12:15 
2011-05-26T13:25 
201 1-0602T1 3:00 
201 1-06-02T08:40 
2011-06-08T13:30 
2011-06-09T08:45 
201 1-06O8T1 1:20 
2011-06-09T09:15 
2011-06-08T12:45 


Session  10:  DTG 
2011-05-24T10:(X) 
2011-05-23T14:15 
201 1-05-24T1 1:00 
201 1-05-31T1 1:45 
2011-05-31T13:45 
2011-05-31T14:00 
2011-05-31T1430 
2011-05-31T18:00 
201 1-06-06T12:50 
201 1-0607T08:45 
201 1-06-09T09:30 
201 1-06-1 3T12:00 
2011-08-14T12:45 
201 1-06-1 3T08:30 
2011-06-13T13:00 


Session  11:  DTG 
2011-05-25T10:00 
2011-05-25T14:20 
201 1-05-25T1 1:40 
2011-06-01T09  35 
2011-06-01T13:45 
2011-06-01T14:15 
201 1-06-02T1 4:35 
201 1-0601T1 3:20 
201 1-06-07T1300 
201 1-06-08T08:45 
201 1-06-10T1 3:00 
201 1-06-14T1 1:45 
2011-06-15T12:10 
201 1-06-13T1 1:40 
2011-06-14T14:35 


Session  12:  DTG 
201 1-05-28T1000 
2011-05-26T14:00 
201 1-05-26T10:45 
201 1-06-02T10:30 
2011-06-02T13:45 
201 1-06-02T1400 
2011-06-03T08:25 
201 1-06O2T1 3:30 
201 1-06-08T09:15 
201 1-06-09T0815 
201 1-06-1 1T12:45 
201 1-06-15T1 1:40 
201 1-06-18T12:00 
201 1-06-14T1 1:30 
2011-06-15T15:50 


Figure  34.  JMP  raw  data:  participant  experiment  times 


Figure  35.  JMP  raw  data:  Sessions  2  and  3  (week  1 ) 
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Figure  36.  JMP  raw  data:  Sessions  5  and  6  (week  2) 
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Figure  37.  JMP  raw  data:  Sessions  8  and  9  (week  3) 
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Figure  38.  JMP  raw  data:  Sessions  1 1  and  12  (week  4) 
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2 

Training 
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99 
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Figure  39.  JMP  raw  data:  total  hits,  FA,  and  clicks 
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B.  CONFIDENCE  DATA 
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Figure  40.  JMP  raw  data:  participant  detection  confidence 
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APPENDIX  I.  CONFIDENCE  AND  THREAT  RAW  DATA 


A.  SME  RATINGS  OF  CHANGES 


The  next  two  tables  show  the  subject  matter  expert  (SME)  ratings  for  all 
changes  in  Scenario  1  and  2.  The  tables  order  the  changes  from  lowest  threat  to 
highest  threat,  ordered  first  by  the  mean  rating,  then  the  lowest  standard 
deviation  of  the  rating.  The  research  team  put  the  changes  into  bins  of  10  “low,” 
“medium,”  and  “high”  threat. 


Scenario  # 

Variable  Name 

SME1 

SME  2 

SME  3 

SME  4 

STDEV 

Average 

1 

burkha gir(1 

2 

2 

2 

2 

r  0.0000 

2 

1 

broken fence no ip 1 

3 

2 

2 

1 

r  0.8165 

2 

1 

ied supply man 1 

1 

2 

1 

4 

k  1.4142 

2 

1 

propane tank sales 1 

2 

2 

3 

2 

r  0.5000 

f 

2.25 

1 

broken wal|1 

2 

3 

2 

2 

0.5000 

2.25 

1 

broken wall 2 

2 

3 

2 

2 

0.5000 

2.25 

1 

new con(ainers 1 

2 

4 

2 

1 

1.2583 

2.25 

1 

poster wal(1 

4 

1 

2 

2 

1.2583 

2.25 

1 

dirt  pile  1 

3 

2 

3 

2 

0.5774 

2.5 

1 

bad miKer 1 

2 

2 

4 

2 

1.0000 

2.5 

1 

ied box 1 

4 

2 

3 

1 

1.2910 

2.5 

1 

broken guardrai|1 

4 

1 

3 

2 

1.2910 

2.5 

1 

reporter 1 

4 

2 

3 

1 

1.2910 

2.5 

1 

bumt truck 1 

4 

3 

3 

3 

0.5000 

3.25 

1 

observer 1 

4 

2 

4 

3 

0.9574 

3.25 

1 

vbied van 1 

4 

3 

2 

4 

0.9574 

3.25 

1 

ied patch 1 

4 

3 

4 

2 

0.9574 

3.25 

1 

tuined house 1 

4 

4 

2 

3 

0.9574 

3.25 

1 

broken w  all ied 1 

2 

3 

4 

4 

0.9574 

3.25 

1 

bad trash pile 1 

1 

4 

4 

4 

'  1.5000 

f 

3.25 

1 

running mar\1 

4 

3 

4 

3 

0.5774 

3.5 

1 

buried tank 1 

4 

4 

2 

4 

1.0000 

3.5 

1 

tire  1 

4 

4 

4 

3 

0.5000 

3.75 

1 

bad curb 1 

4 

3 

4 

4 

0.5000 

3.75 

1 

bad bag pile 1 

4 

3 

4 

4 

0.5000 

3.75 

1 

concrete ied 1 

4 

4 

4 

3 

0.5000 

3.75 

1 

odd drums 1 

4 

4 

4 

4 

0.0000 

4 

1 

bad rubble 1 

2 

4 

5 

5 

1.4142 

4 

1 

vid camera 1 

4 

4 

4 

5 

0.5000 

4.25 

1 

bad_barrels_1 

4 

4 

5 

5 

0.5774 

4.5 

Table  16.  SME  ratings  for  Scenario  1  changes 
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Scenario  * 

Variable  Name 

SME  1 

SME  2 

SME  3 

SME  4 

STDEV 

Average 

2 

ice cream dude 

1 

1 

1 

2 

0.5000 

1.25 

2 

new tractor 

1 

1 

2 

1 

0.5000 

'  725 

2 

nobody ia cp 

2 

1 

1 

1 

0.5000 

'  725 

2 

new van 

2 

2 

1 

1 

0.5774 

75 

2 

jingle truck 

1 

1 

3 

1 

1.0000 

75 

2 

weird-dudes 

1 

3 

2 

1 

0.9574 

775 

2 

new cars 

2 

1 

3 

1 

0.9574 

775 

2 

press man 

2 

2 

2 

2 

h  0.0000 

2 

2 

van bomb 

2 

3 

2 

[  0.8165 

2 

2 

bomb mal<er 

4 

2 

1 

1 

1.4142 

2 

2 

truck-wreck 

2 

3 

2 

2 

0.5000 

2.25 

2 

new  boxes no-phone bool 

2 

2 

3 

3 

0.5774 

2.5 

2 

UN sacks 

4 

3 

2 

2 

0.9574 

2.75 

2 

broken,  wall 

3 

2 

2 

4 

0.9574 

2.75 

2 

new rubble no kids 

2 

3 

4 

2 

0.9574 

2.75 

2 

back to back vans 

4 

3 

3 

2 

0.8165 

3 

2 

new-car open street 

3 

4 

3 

2 

0.8165 

3 

2 

propane.tanks 

2 

3 

4 

3 

0.8165 

•j 

2 

boarded up shop 

2 

2 

4 

4 

1.1547 

3 

2 

<urned truck 

4 

3 

3 

3 

0.5000 

3.25 

2 

walLtruck 

4 

2 

4 

3 

0.9574 

3.25 

2 

bacLcart 

4 

3 

4 

3 

0.5774 

3.5 

2 

bad pile 

2 

4 

4 

4 

1.0000 

3.5 

2 

new barrels 

4 

4 

4 

2 

1.0000 

3.5 

2 

new  rock 

4 

3 

4 

4 

0.5000 

3.75 

2 

dead body 

5 

3 

4 

3 

0.9574 

3.75 

2 

fridge no play 

4 

4 

4 

5 

0.5000 

4.25 

2 

bomb car 

4 

4 

4 

5 

0.5000 

4.25 

2 

suicide bomber 

5 

5 

5 

4 

0.5000 

4.75 

2 

ip_suicide_bomber 

5 

5 

5 

5 

0.0000 

5 

Figure  41 .  SME  ratings  for  Scenario  2  changes 


B.  PARTICIPANT  DETECTION  CONFIDENCE 


, 

| 

I 

0007 

0490 

0891 

1133 

2888 

3767 

4814 

9767 

LOW 

Med 

LOW 

Med 

High 

LOW 

LOW 

Med 

High 

Low 

Med 

LOW 

Med 

High 

low 

Med 

High 

LOW 

Med 

Session  3 

5 

Session  3 

Session  3 

6 

0 

Session  3 

0 

, 

Session  3 

26 

0 

Session  3 

. 

Session  3 

. 

■ 

Session  3 

2 

5 

Session  6 

1 

1 

13 

Session  6 

: 

? 

9 

Session  6 

3 

0 

Session  6 

1 

1 

18 

Session  6 

10 

1 

7 

Session  6 

/ 

5 

16 

Session  6 

i 

1 

70 

Session  6 

4 

3 

IS 

Session  9 

6 

0 

14 

Session  9 

3 

3 

17 

Session  9 

0 

0 

Session  9 

0 

0 

22 

Session  9 

5 

0 

19 

Session  9 

2 

4 

21 

Session  9 

0 

2 

21 

Session  9 

3 

0 

21 

Session  12 

1 

0 

15 

Session  12 

2 

6 

12 

Session  12 

0 

0 

28 

Session  17 

0 

0 

19 

Session  17 

5 

3 

12 

Session  12 

2 

2 

23 

Session  12 

0 

2 

25 

Session  12 

4 

0 

71 

Totals 

9  3  47 

12  14  43 

9  0 

92  1  9  68 

46  4  38 

13  15  60  4  12  74 

13  8  63 

1407 

1822 

2458 

4395 

5669 

6849 

9862 

LOW 

Med 

Low 

Med 

Low 

Med 

LOW 

Med 

Low 

Med 

LOW 

Med 

Med 

Session  3 

9 

sess.on  3 

0 

9 

Session  3 

6 

10 

Session  3 

0 

10 

Session  3 

• 

■ 

■ 

Session  3 

12 

Session  3 

‘ 

9 

Session  6 

1 

7 

1  • 

Session  6 

0 

0 

18 

Session  6 

2 

Session  6 

• 

0 

13 

Session  6 

6 

5 

6 

Session  6 

: 

: 

20 

Session  6 

I 

4 

15 

Session  9 

3 

6 

17 

Session  9 

0 

0 

24 

Session  9 

2 

3 

19 

Session  9 

0 

3 

12 

Session  9 

2 

: 

14 

Session  9 

0 

0 

2  ' 

Session  9 

2 

5 

17 

Session  12 

1 

2 

24 

session  12 
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Figure  42.  Confidence  raw  data  by  raw  data 
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APPENDIX  J.  DESIGNING  AND  EXECUTING  CHANGE 
DETECTION  SCENARIO  TRAINING 


A.  SCENARIO  INSTRUCTIONS  FOR  EDITORS 

•  Install  VBS2  VTK  1 .40  or  better. 

•  Label  the  “C,”  “V,”  and  “B”  keys  of  the  keyboard  “H,”  “M”  and  “L”  for 
ease  of  recognition  by  the  participant. 

•  Open  VBS2  by  double-clicking  on  the  desktop  shortcut  called 
“VBS2  Administrator.” 

•  In  the  bottom  right  of  the  screen,  where  it  says  “Profile,”  click  on  the 
name,  then  click  “New,”  then  create  a  new  unique  profile  name. 
Click  “Ok.” 

•  In  the  bottom  left  of  the  screen,  click  the  “Options”  button.  In  the 
pop-up  menu  that  appears,  select  “Video  Options.”  Set  the  native 
resolution  and  aspect  ratio  for  the  monitor  to  be  used.  Click  “Ok” 
when  finished. 

•  Exit  VBS2. 

•  Open  the  profile.txt  file  included  in  the  archive  on  the  DVD.  Select 
the  entire  text  and  copy  to  the  clipboard.  (Edit~>Select  All  then 
Edit— >  Copy) 

•  Navigate  to  the  location  where  the  unique  profile  (in  4  above)  was 
created.  For  Win7,  it  will  either  be  in  ...Documents/VBS2  or  in 
...Documents/VBS2  Other  Profiles  ,  depending  on  whether  this  is 
the  first  profile  to  have  been  set  up  on  VBS2  on  the  computer  being 
used.  Open  the  profile  file  with  notepad,  delete  all  the  text  in  the 
file,  then  paste  in  all  the  text  copied  in  step  7  above.  Save  the 
profile  file.  This  ensures  all  settings  are  as  intended  for  the  change 
detection  training.  (For  WinXP,  these  folders  will  be  located  in  “My 
Documents”) 

•  Open  the  archive  from  the  DVD.  Extract  or  copy  the  mpmissions 
folder  to  the  same  location  as  the  unique  profile  file.  Again,  this 
location  will  vary  depending  on  whether  this  is  the  first  profile 
created  on  the  system  or  not.  If  the  profile  is  the  first  profile  to  be 
created,  this  location  will  be  ...Documents/VBS2  as  seen  in  Figure 
43. 
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Path  if  this  profile  is  the  first  in  the  system 

►  libraries 

Documents  ►  VBS2  ► 

Organize  *  Share  with 

•  Bum  New  folder 

4  Favorites 

■  Desktop 

Documents  library 

VBS2 

Extracted  folder  containing  mission  folders 

4  Downloads 

Name 

/  Date  modified 

Type 

1  Dropbox 

..  Recent  Places 

i  AAR  / 

3/17/2011  242  PM 

File  lolder 

i  Config  / 

3A0/2011  8:16  PM 

File  folder 

libraries 

A.  missions  / 

12/71/2010 12:49 . 

File  folder 

Documents 

i  mpmissions 

8/31/20116-30  AM 

File  folder 

i  Music 

A  Saved 

3/16/2011  330  PM 

File  folder 

k  Screen 

8/30/2011  12:03  PM 

File  folder 

%  Videos 

l  UserSaved 

3/11/2011  6:48  AM 

File  folder 

Mike  < - 

Profile  file,  named  by  unique  name 

V8S2 PROFILE  File 

Homegroup 

•  VBS2 

8/31/2011  635  AM 

CFG  File 

Figure  43.  File  view 


If  the  profile  is  not  the  first  in  the  system,  then  refer  to  Figure  44. 
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Figure  44.  File  view  without  a  profile 


•  The  mission  folders  in  ...Impmissions  should  include: 
Session_1  .map_samawah50km_ieed 
Session_2.map_samawah50km_ieed 
Session_3.map_samawah50km_ieed 
Session_4.map_samawah50km_ieed 
Session_5.map_samawah50km_ieed 
Session_6.map_samawah50km_ieed 
Session_7.map_samawah50km_ieed 
Session_8.map_samawah50km_ieed 
Session_9.map_samawah50km_ieed 
Session_1 0.map_samawah50km_ieed 
Session_1 1  .map_samawah50km_ieed 
Session_1 2_T  raining. map_samawah50km_ieed 
Session_12_Non_T raining. map_samawah50km_ieed 
T  rainingSession_1  A.map_samawah50km_ieed 
TrainingSession_1  B.map_samawah50km_ieed 
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TrainingSession_2A.map_samawah50km_ieed 

Training2Session_2B.map_samawah50km_ieed 


Open  VBS2  by  double-clicking  on  the  desktop  shortcut  called 
“VBS2  Administrator.” 

Select  “Mission  Editor”  from  the  main  menu. 

Select  “As  Samawah  [GAA}”  and  click  “OK.” 

Once  the  terrain  has  loaded,  the  2-D  editor  map  will  be  presented. 

The  system  is  now  in  edit  mode.  Any  changes  will  need  to  be 
saved  and  will  overwrite  the  provided  missions  unless  saved  by  a 
different  name. 

In  the  top,  left  corner  of  the  screen  click  File  then  Load. 

For  the  first  session,  select  the  appropriate  scenario.  Click  “OK.” 

Ensure  the  participant  is  wearing  headphones  or  using  speakers  at 
an  appropriate  volume  level  and  sitting  comfortably. 

From  the  same  “File”  menu,  select  “Preview.” 

The  participant  should  follow  all  instructions  given  in  the  scenario 
which  features  voiceover  instructions. 

At  the  conclusion  of  the  scenario,  press  the  “ESC”  key.  Click 
“Abort.”  Now,  additional  scenarios  can  be  loaded  and  run  in  the 
same  manner.  However,  it  is  critical  that  any  data  files  be  extracted 
as  follows  in  the  next  step  BEFORE  running  another  scenario.  If 
data  is  not  extracted  before  running  another  scenario,  it  will  be 
overwritten. 

All  scenarios,  except  the  training  scenarios,  will  write  to  data  files  in 
the  actual  VBS2  root  directory.  If  VBS2  was  installed  to  the  C  drive 
using  the  default  settings,  this  path  will  be  C:\Bohemia 
lnteractive\VBS2.  Table  17  shows  which  data  files  should  be 
obtained  from  this  directory  immediately  after  a  participant  has 
finished. 
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Scenario 

headDirection.txt 

startTimes.txt 

clicked.txt 

targetFinalScore.txt 

targetsFound.txt 

Survey.txt 

1,4,7,10 

V 

2,5,8,11 

V 

V 

V 

V 

3,6,9 

V 

V 

V 

V 

V 

12 

V 

V 

V 

V 

V 

V 

Training 

Table  17.  Data  file  matrix 

B.  HOW  TO  DESIGN  A  CHANGE  DETECTION  SCENARIO 

Change  Detection  Scenario  Development  Checklist: 


Windows  7  was  used  for  these  instructions.  Windows  XP  or  Windows 
Vista  will  have  similar  operations  but  may  not  exactly  match  these  instructions  for 
the  folder  locations. 


1 .  VBS2  custom  scenario  designs  are,  by  default,  saved  to  the  folder: 

My  Documents/VBS2/mpmissions/scenar/oA/ame 

2.  Within  this  folder,  there  are  several  key  files.  When  a  scenario  is 
saved  within  VBS2,  this  scenario  folder  will  be  created  in  the  mpmissions  folder. 
The  files  mission. sqf,  mission. biedi,  and  mission. sqm  will  automatically  be 
created  within  this  folder.  There  is  no  need  to  ever  open  any  of  these  mission 
files  when  designing  this  scenario.  To  create  a  custom  scenario,  the  following 
files  will  be  added  to  this  scenario  folder: 

init.sqf  -  init.sqf  is  the  script  file  which  will  be  run  automatically  at  the 
beginning  of  any  scenario.  Any  procedural  scripts  written  in  this  text  file  will  be 
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executed  once  at  the  beginning  of  the  scenario.  However,  a  “do  while”  structure 
can  be  used  in  the  init.sqf  file  to  repeat  a  recurring  operation  for  the  duration  of 
the  scenario.  For  this  scenario,  this  file  is  used  to  bind  keyboard  commands  to 
external  script  file  execution,  create  data  files,  conduct  voiceover  instruction  at 
the  beginning  of  a  scenario,  and  begin  a  “do  while”  loop  for  tracking  head 
direction  of  the  participant. 

description. ext  -  description. ext  can  be  used  to  customize  the  interface, 
but  is  also  important  for  defining  the  sound  files  to  be  used  in  the  scripting.  For 
this  scenario  we  will  use  it  strictly  to  define  sound  files  to  be  used  in  the 
voiceover  instructions. 

.sqf  -  A  .sqf  file  is  an  external  script  file  that  will  execute  on  command. 
For  this  scenario  design,  we  will  write  several  .sqf  files  which  will  be  bound  to 
keyboard  commands.  When  a  participant  presses  the  respective  key,  the  bound 
.sqf  file  will  be  executed.  These  .sqf  files  should  all  be  created  with  unique 
names  and  saved  within  the  same  scenario  folder  as  init.sqf  and  mission. sqf. 

3.  To  begin  creating  the  baseline  scenario,  open  VBS2  by  using  the 
“VBS2  Administrator”  executable. 

4.  On  the  Main  Menu,  select  “Mission  Editor.”  From  the  list  of  terrain, 
choose  “As  Samawah  [GAA]”  and  click  “OK.” 

5.  After  the  terrain  database  loads,  you  will  be  presented  with  the 
scenario  editor  map.  By  holding  down  the  right  mouse  button  (RMB),  the  map 
can  be  scrolled  when  dragging  the  mouse.  Use  the  mouse  wheel  to  zoom  in  and 
out.  For  the  design  of  this  scenario,  begin  at  the  intersection  at  map  coordinate 
3355  8900.  The  map  coordinates  are  listed  as  on  a  standard  military  map,  with 
vertical  and  horizontal  lines.  The  first  four  numbers  listed  are  the  horizontal 
coordinates.  The  last  four  digits  are  the  vertical  coordinates.  The  numbers  can 
be  seen  at  the  extreme  top  and  extreme  left  of  the  map. 
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6.  First,  create  the  HMMWV  at  the  intersection  by  selecting  the  “Vehicle” 
tab  (NOT  the  “Empty  Vehicle”)  in  the  lower  right  of  the  screen  (If  this  is  not 
visible,  click  the  two-way  arrow  in  the  lower  right  of  the  screen  to  expand  the 
panel). 
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Figure  45.  HMMWV  creation 


7.  From  the  list,  expand  the  “U.S.  Army  Car  Desert”  sub-list.  Select 
“Ml  114  HMMWV  -M2”  from  the  list.  Assign  the  vehicle  the  variable  name 
“veh_1”  by  typing  in  the  “Name”  block  as  seen  in  Figure  41.  Under  “Has  Crew?  / 
Special”  select  “true,”  “None,”  and  “Player  as  Gunner”  as  seen  in  Figure  45. 
Click  “OK.” 

8.  To  enter  a  3-D  view  of  the  selected  object  (In  this  case  the  HMMWV), 
press  the  “M”  key  on  the  keyboard.  Once  in  the  3-D  editor,  use  the  following 
commands  to  control  the  view  and  manipulate  objects: 

W,  A,  S,  D  -  Horizontal  translation  of  viewpoint 

G,Z  -  Vertical  translation  of  viewpoint  (Hint:  higher  viewpoints  allow  faster 
transition  across  the  map). 
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Left-Shift  -  Use  in  combination  with  translation  keys  to  move  quickly 
around  the  map. 


Hold  RMB  and  move  mouse  -  View  rotation 


Hold  Left-Alt  -  Translation  interface,  click  and  drag  on  respective  arrows  to 
move  objects  in  X-Y-Z  plane. 

Hold  Left-Alt  +  Spacebar  -  Rotation  Interface,  Click  and  drag  on  respective 
circles  to  rotate  objects  in  the  X-Y-Z  plane. 


9.  Using  the  commands,  position  the  vehicle  so  it  is  facing  northwest  on 
the  road. 


Figure  46.  Placing  the  vehicle 


10.  Now,  create  the  route  for  the  vehicle.  Select  the  vehicle  and  click  the 
RMB.  From  the  pop-up  menu,  select  “Assign  New  Waypoint.” 
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Figure  47.  Selecting  from  the  pop-up  menu 

1 1 .  Extend  the  waypoint  to  the  end  of  the  current  road  where  it  creates  a 
“T”  with  the  main  road  (the  red  circle  in  Figure  47).  Click  the  Left  Mouse  Button 
(LMB). 


Figure  48.  Waypoint  object  properties 
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12.  In  the  Object  Properties  box,  select  “MOVE”  for  type,  “CUSTOM”  for 
Speed  (km/h),  and”10”  in  the  box  to  the  right  (See  Figure  48).  This  is  Waypoint  1 
for  the  vehicle.  Click  OK. 

13.  All  editor  operations  can  be  performed  in  the  3-D  Editor  or  2-D  Editor 
interface. 

14.  Press  the  “M”  key  to  return  to  the  2-D  Editor. 

15.  Select  the  square  that  represents  the  newly  created  waypoint.  Click 
the  RMB  to  bring  up  the  Pop-Up  menu.  Select  “Add  New  Waypoint,”  extend  the 
new  waypoint  and  place  it  at  the  intersection  in  the  vicinity  of  coordinate  3380 
8965. 

16.  Use  the  same  procedure  described  above  to  place  two  additional 
waypoints  at  the  locations  specified  with  red  circles  in  Figure  49. 


Figure  49.  Adding  Waypoints  3  and  4 


17.  Now  save  this  scenario  by  clicking  File  -->Save.  In  the  dialog  box, 
enter  the  title  of  this  scenario  in  both  the  “Mission”  and  “Title”  text  boxes  as 
shown  in  Figure  49.  Click  OK. 
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Figure  50.  Saving  the  scenario 


18.  Now  ALT -TAB  to  return  to  the  Windows  desktop.  Open  the  folder 
“My  Documents/VBS2/mpmissions”  and  ensure  that  the  “baseline”  scenario 
folder  has  been  created.  It  will  be  a  folder  named 

“baseline. map_samawah50km_ieed.”  See  Figure  51. 
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Figure  51 .  Scenario  folder  location 

19.  Open  the  “baseline. map_samawah50km_ieed”  folder.  There  is  no 
init.sqf  or  description. ext  present  in  the  folder. 

20.  Using  the  Windows  Start  menu,  open  the  basic  windows  Notepad 

application.  Click  File  ->  Save  As.  Navigate  to  the 

“baseline. map_samawah50km_ieed”  folder.  Select  “All  Files”  from  the  “Save  as 
type”  dropdown  and  type  init.sqf  in  the  “File  Name”  text  box  (See  Figure  52). 

21 .  Use  the  same  procedure  in  (20)  to  create  the  description. ext  file  in  the 
scenario  folder. 
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Figure  52.  Creating  the  init.sqf  file 


22.  In  the  scenario  folder,  create  a  new  blank  folder  and  name  it 
“SOUND.” 

23.  When  complete,  the  scenario  folder  contents  should  look  like  Figure 
53. 


Name 

SOUND 
I  description 
1 3"j  init 

| _ |  mission. biedi 

I  mission 
| _ !  mission. sqm 


Figure  53.  Complete  scenario  folder 


24.  Scripts  will  be  written  to  these  blank  files  later.  For  now,  return  to 

VBS2. 
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25.  To  test  the  vehicle  waypoints,  run  the  simulation  by  clicking  File  — > 
Preview.  The  vehicle  will  stop  at  each  waypoint.  Press  1  then  9  on  the  keyboard 
to  tell  the  Al  driver  to  proceed  to  the  next  waypoint.  After  verifying  that  the 
vehicle  navigates  correctly,  press  the  ESC  key,  then  click  Abort  to  return  to  the 
editor.  If  there  are  issues  with  vehicle  navigation,  use  the  editor  to  click  on  the 
waypoint  and  drag  to  adjust  to  a  new  location.  Keep  running  the  simulation  until 
navigation  occurs  without  any  glitches. 

26.  The  next  step  is  to  create  all  ambient  vehicle  traffic.  Vehicle  traffic  will 
be  created  to  run  in  loops  around  circuits  around  the  scenario  area.  Ambient 
vehicle  traffic  should  not  share  the  same  road  as  the  participant  vehicle  unless  it 
is  at  least  a  four  lane  road.  On  two-lane  roads  and  smaller,  ambient  vehicle 
traffic  can  cause  the  participant’s  vehicle  to  make  bad  routing  decisions.  First, 
create  a  vehicle  at  the  intersection  by  selecting  the  “Vehicle”  tab  (NOT  the 
“Empty  Vehicle”)  in  the  lower  right  of  the  screen  (If  this  is  not  visible,  click  the 
two-way  arrow  in  the  lower  right  of  the  screen  to  expand  the  panel).  For  this 
example,  we  will  choose  a  city  bus.  Expand  the  IQ  Civilian  Vehicle  listing  and 
select  the  bus  as  seen  in  figure  54.  This  does  not  need  a  specific  variable  name 
assigned.  Click  OK. 
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Figure  54.  Selecting  the  Bus  for  ambient  traffic 


27.  To  create  a  loop,  assign  a  series  of  waypoints  using  the  RMB  pop-up 
menu  and  selecting  “Add  New  Waypoint.”  Add  a  series  of  waypoints  in  this 
manner  ensuring  the  last  waypoint  is  near  the  vehicle’s  starting  position.  To 
cause  the  looping  behavior,  use  the  RMB  menu  on  the  final  waypoint,  select 
“Assign  Next  Waypoint,”  and  then  LMB  click  on  the  first  waypoint  assigned. 
Each  vehicle  added  requires  another  run  of  the  simulation  for  testing.  Testing 
often  prevents  unknown  problems  from  creeping  in. 

28.  For  this  scenario,  three  aircraft  were  added  to  the  simulation  for 
ambient  activity.  Create  a  Blackhawk  helicopter  at  about  319  881.  Enter  the 
parameters  shown  in  Figure  11:  “hel_1”  for  “Name,”  “hel_1  flylnHeight  150;”  in 
“Initialization  Statements,”  and  “Flying”  in  the  “Special”  text  box.  Click  OK. 
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Figure  55.  Creating  the  Blackhawk 

29.  Click  the  RMB  on  the  created  aircraft  and  select  “Add  New  Waypoint” 
and  position  the  waypoint  somewhere  near  the  aircraft.  Select  “Speed  Mode”  as 
“Normal”  and  “Aircraft  Height”  as  “300”  and  “AGL”  as  seen  in  Figure  56.  Click 
OK.  Add  two  additional  waypoints  so  that  the  helicopter  will  approach  near  the 
participant  during  the  scenario.  In  the  experiment  scenario,  the  helicopter 
crosses  as  the  participant’s  vehicle  approaches  the  first  turn  (about  the  4  minute 
mark)  and  about  3  minutes  later  while  traveling  down  the  long  Route  Red.  Since 
this  requires  some  synchronization,  a  trigger  must  be  placed  to  tell  the  aircraft  to 
proceed  to  the  second  waypoint.  Create  the  trigger  synchronization  as  follows: 

a.  Select  “Trigger”  in  the  quick  menu  at  the  lower  right  side  of  the 
editor.  Double-click  the  LMB  which  will  bring  up  the  trigger  dialogue  box.  Enter 
the  parameters  as  seen  in  Figure  56.  Click  OK.  This  will  create  a  rectangular 
area  which  will  trigger  a  switch  when  any  friendly  unit  passes  into  the  area.  It  has 
a  dimension  of  8  x  8  meters  in  the  simulation.  It  is  named  helo_trigger. 

b.  This  created  trigger  will  be  used  to  tell  the  helicopter  to  proceed 

to  the  next  waypoint.  So,  select  the  first  waypoint  near  the  aircraft,  click  the 
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RMB,  and  in  the  menu  choose  “Sync  to  Trigger.”  Click  on  the  newly  created 
trigger. 


c.  Run  the  simulation  and  adjust  as  needed  to  display  proper 
behavior  to  the  participant. 


Figure  56.  Assigning  Blackhawk  Waypoint  1 
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Figure  57.  Creating  a  synchronization  trigger 


30.  Using  the  same  method,  create  the  flight  of  two  AH-64  helicopters  so 
that  they  pass  the  participant  when  they  reach  the  “Victory  Bridge”  landmark. 
Continue  to  run  tests  for  synchronization. 

31.  Create  several  sound  triggers  throughout  the  route.  Using  the  same 
idea  as  the  synchronization  triggers  for  aircraft,  the  sound  triggers  will  be 
configured  to  play  a  specific  sound  when  the  participant  enters  the  trigger  area. 
To  create  a  sound  trigger  in  the  editor,  press  F7  or  select  “Trigger”  in  the  quick 
menu  at  the  lower  right  side  of  the  editor.  Double-click  on  the  map  to  place  the 
trigger  at  the  desired  location.  Fill  out  the  dialog  box  to  match  Figure  57.  This 
will  create  a  trigger  that  will  play  the  prayer  call  ambient  sounds  when  any 
friendly  unit  enters  the  10  x  10  area. 
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OBJECT  PROPERTIES 
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Figure  58.  Dialog  box  for  a  Prayer  Call 


32.  Run  tests  to  ensure  the  sounds  trigger  as  desired.  Remember,  any 
friendly  unit  can  trigger  these  triggers,  so  pay  attention  to  movement  of  all 
friendly  units,  including  Iraqi  police  or  soldiers,  that  it  does  not  accidentally  trigger 
a  sound. 

33.  There  are  several  different  ways  that  the  research  team  introduced 
people  into  the  scenario.  Some  are  just  static  placements;  they  never  move. 
Others  have  pre-defined  looping  routes.  Some  have  routes  with  triggered  starts 
while  still  others  start  moving  when  the  scenario  begins. 

34.  To  create  a  group  of  children  that  continue  running  back  and  forth, 

create  five  children  at  approximately  grid  3352  8907.  Press  FI  or  select  “Unit” 
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form  the  quick  menu  in  the  lower  right  corner  of  the  editor.  Double-click  on  the 
map  to  select  the  position  of  the  child.  Expand  the  “IQ  Civilians”  list  and  select 
one  of  the  children  available.  Add  about  five  children  in  this  manner. 
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Figure  59.  Selecting  a  child 


Figure  60.  Waypoint  map  of  children  playing 
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35.  To  cause  the  children  to  appear  to  be  playing,  create  several 
waypoint  cycles.  For  each  child  create  a  set  of  two  waypoints.  The  first  waypoint 
should  be  of  type  “MOVE.”  The  second  waypoint  should  be  of  type  “CYCLE.” 
This  will  cause  a  looping  behavior  that  will  resemble  playing. 

36.  To  create  a  person  who  will  move  only  when  the  participant  reaches  a 
certain  part  of  the  map,  a  trigger  and  a  waypoint  synced  to  that  trigger  will  again 
need  to  be  created. 

37.  To  create  a  small  group  of  men  quickly,  select  from  the  Group  in  the 
quick  menu  or  press  F2. 

38.  To  create  a  person  who  moves  when  the  simulation  begins,  simply 
add  a  series  of  waypoints.  The  person  will  automatically  proceed  to  the  first 
waypoint  when  the  simulation  begins. 

39.  Static  people  are  added  just  like  objects. 

40.  When  adding  any  person  in  VBS2,  they  will  appear  to  be  holding  a 
weapon.  However,  unless  they  are  police  or  Army,  they  will  not  appear  to  hold  a 
weapon  once  the  simulation  begins. 

41 .  It  is  critical  that  as  people  are  added  that  the  simulation  is  tested  each 
time  for  possible  problems  introduced  by  the  new  units  in  the  scenario. 

42.  Next,  place  objects  in  the  baseline  scenario  which  we  can  change  or 
manipulate  for  the  change  scenarios. 

43.  For  any  voiceovers  in  the  simulation,  record  .wav  files.  The  best 
freeware  program  to  do  this  is  called  Audacity.  Audacity  is  available  at 
http://audacity.sourceforqe.net/. 

44.  To  use  Audacity  to  create  a  .wav  file,  follow  the  following  steps: 

a.  Step  1 

b.  Step  2 
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45.  In  order  to  use  the  sounds  in  VBS2,  they  must  be  configured  to  work 
with  the  scenario  using  the  description. ext  file.  Before  this  is  written,  ensure  that 
all  created  .wav  files  to  be  used  in  the  scenario  are  moved  to  the  SOUND  folder 
in  the  scenario  folder.  The  format  for  writing  the  description. ext  is  as  follows: 


class  CfgSounds 

{ 

sounds[]  =  {}; 
class  soundNamel 
{ 

name  =  “  soundName  1 

sound[]  =  {“  soundNamel . wav ,”1, 1}; 

titles[]  =  {}; 

}; 

class  soundName2 

{ 

name  =  “  soundName  2”; 
sound[]  =  {“  sour)dA/a/77e2.wav,”1,1}; 
titles[]  =  {}; 

}; 

}; 


Although  it  is  not  required  to  use  the  same  name  in  the  sound  definitions, 
consistency  is  recommended.  Once  this  description. ext  is  created,  the  following 
VBS2  scripting  command  would  play  the  wave  file  soundNamel  in  the 
simulation: 


playSound  “soundNamel”; 

46.  The  init.sqf  for  the  baseline  scenario  is  fairly  simple. 

_blank  =  //create  a  blank  string  for  writing  to  file 

_filename2  =  “headDirection.txt”;  //create  a  variable  for  the  filename 

_append  =  format[“Write(%1  )@%2,”_filename2,_blank];  //format  the  file  to  be 
written 

_writeFile  =  pluginFunction  [“VBSPIuginFileAccess”  ,  _append];  //write  the  file  to 
the  specified  text  file 

hint  “This  is  a  test.”;  //display  the  dialog  “This  is  a  test”  on  the  screen  for  the 
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pareticipant 

playSound  “test”;  //play  the  voiceover  wav  file  associated  with  “test”  in 
description. ext 

sleep  5;  //pause  5  seconds  before  executing  the  next  line.  This  should  be  at 
least  as  long  as  the  //previous  .wav  file  to  avoid  crossover, 

hint  “This  is  test2”;  //display  the  dialog  “This  is  a  test2”  on  the  screen  for  the 
pareticipant 

playSound  “test2”;//play  the  voiceover  wav  file  associated  with  “test2”  in 
description. ext 

sleep  5;  //pause  5  seconds  before  executing  the  next  line.  This  should  be  at 
least  as  long  as  the  previous  //.wav  file  to  avoid  crossover 
while  {alive  player}  do  //begin  execution  of  a  do  while  loop  that  continues  while 
the  player  is  still  alive. 

//it  will  write  the  relative  head  direction  of  the  participant  every  0.2 
seconds  (adjust  sleep 

//parameter  to  changejon  a  new  line  in  the  file  headDirection.txt 
including  current  sim  time 
{ 

_headDir  =  getDir  player; 

_vehDir  =  getDir  veh_1 ; 

_relDir  =  _headDir  -  _vehDir; 

_time  =  time; 

_append  =  format[“AppendLine(%1  )@%2,%3,”_filename2,_relDir,_time]; 
_writeFile  =  pluginFunction  [“VBSPIuginFileAccess”  ,  _append]; 
sleep  0.2; 

}; 


47.  When  all  objects  and  triggers  have  been  placed  in  the  base  scenario, 
test  it  several  times  to  ensure  consistency  in  performance.  If  any  anomalous 
behavior  presents  itself,  work  back  from  the  last  change  and  run  again. 
Sometimes  adjusting  the  waypoint  of  the  HMMWV  route  slightly  will  rectify  the 
issue. 

48.  Save  the  baseline  scenario. 

49.  With  the  baseline  scenario  open  in  the  2-D  editor,  save  the  scenario 
again  as  “Change_1 .”  This  provides  the  baseline  as  a  model  for  creating  the  first 
change  scenario. 

50.  Now  remove  and  place  objects  as  necessary  to  create  the  first 

scenario.  It  is  important  to  name  objects  that  will  be  identified  as  a  change 
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because  that  variable  name  will  be  added  to  the  array  which  lists  all  the  changes 
in  the  scenario. 

51.  Create  three  blank  external. sqf  files.  Using  the  Windows  Start  menu, 
open  the  basic  windows  Notepad  application.  Click  File  ~>  Save  As.  Navigate 
to  the  “baseline. map_samawah50km_ieed”  folder.  Select  “All  Files”  from  the 
“Save  as  type”  dropdown  and  type  low.sqf  in  the  “File  Name”  text  box  (See 
Figure  8).  Using  the  same  method,  create  two  additional  .sqf  files  named  med.sqf 
and  high. sqf. 

52.  Bind  keys  in  init.sqf.  Open  the  init. sqf  file  and  write  the  following  lines 
of  code  to  bind  key  presses  to  the  external  .sqf  files  just  created.  Using  the  DIK 
code  reference,  determine  the  codes  for  which  keys  to  bind.  The  following  codes 
were  used  in  the  experiment: 

DIK_C  0x2E 
DIK_V  0x2  F 
DIK_B  0x30 

The  script  to  bind  the  key  to  execution  of  the  appropriate  external  script  file  is  as 
follows: 

0x2E  bindKey  “nul  =  []  execVM  low.sqf”;  //binds  the  “C”  key  to  execution 
of  low.sqf. 

0x2F  bindKey  “nul  =  []  execVM  ‘med.sqf’”;  //binds  the  “V”  key  to  execution 
of  med.sqf. 

0x30  bindKey  “nul  =  []  execVM  ‘high. sqf”;  //binds  the  “B”  key  to  execution 
of  high. sqf. 

53.  In  init.sqf,  instantiate  the  array  of  objects  to  be  identified  as  changes 
in  the  scenario.  Use  the  variable  names  set  when  creating  the  object  in  the 
editor.  An  example  array  for  three  changes  in  a  scenario  would  be  instantiated  in 
init.sqf  as  follows: 

changeArray  =  [changel,  change  2,  change3]; 

54.  Now  write  the  external  script  files.  Start  with  low.sqf  and  then  take 
advantage  of  copy  and  paste  to  write  med.sqf  and  high. sqf.  The  code  for  a  script 
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file  which  looks  for  an  changes  listed  in  the  changeArray  for  a  distance  of  60 
meters  and  within  a  15  degree  azimuth  follows: 


totalClicks  =  totalClicks  +  1 ;  //add  to  tally  of  total  presses 
clickL  =  clickL  +  1 ;  //add  to  tally  of  low  confidence  presses 
_1  click  =  “lowConf”;  //create  a  string  variable  for  data  output 
_blank  =  //create  a  string  variable  for  blank  lines 
_headDir  =  getDir  player; 

_vehDir  =  getDir  veh_1 ; 

_relDir  =  _headDir  -  _vehDir;  //compute  head  direction  relative  to  vehicle  front  at 
button  press  time 

_time  =  time;  //get  sim  time  at  time  of  button  press 
_filename  =  “targetsFound.txt”;  //name  the  text  file  to  append 
_filename3  =  “clicked.txt”;  //name  another  text  file  to  append 
_append  = 

format[“AppendLine(%1  )@%2,%3,%4,”_filename3,_1  click, _relDir,_time];  //record 
button  //press 

_writeFile  =  pluginFunction  [“VBSPIuginFileAccess”  ,  _append];  //write  to  file 

{ 

if  ((player  isLookingAt  Lx.  1 5])  &&  ((player  distance  _x)  <  60))  then{  //check 
60  meters  and  15  degree 

_filename  =  “targetsFound.txt”;  //string  variable  for  data 
_confidence  =  “confl  correct”;  //string  variable  for  data 
scoreL  =  scoreL  +  1 ;  //add  to  low  confidence  correct 
totalCorrect  =  totalCorrect  +  1 ;  //add  to  total  correct 
_append  = 

format[“AppendLine(%1)@%2,%3,%4,%5,”_filename,_x,_relDir,_time,_1  click]; 
_writeFile  =  pluginFunction  [“VBSPIuginFileAccess”  ,  _append];  //write  to  file 
playSound  “dinger”;  //play  ding  sound  for  knowledge  of  results 
}; 

}forEach  changeArray;  //check  for  each  item  in  the  changeArray 

55.  Run  the  scenario.  Test  low,  medium,  and  high  confidence  levels. 
Check  data  files.  All  data  files  will  write  to  the  VBS2  root  folder  located  at 
C:/Bohemia  lnteractive/VBS2. 

56.  To  create  another  change  scenario,  open  the  baseline  scenario  and 
then  save  under  a  new  name  such  as  “Change2” 
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APPENDIX  K.  CHANGES  USED  IN  THIS  EXPERIMENT 


A.  SCENARI0 1 


5CENARQ1 

1.  t  'C_l 

2.  Du*nt-trjcr_l 

5.  3 *coa '<_t : - «_l 
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S.  ,“-r  ‘ 

r  «a_DCW_l 

10  _pnan_l 
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12.  vCred_van_l 
13  «0_34t£T._l 
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13  "j 

16  D-oce-i_*e  _2 

17  -«_1 
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22  39C_Me_3  f_l 
22 
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26  bu'«a_tarA_i 

27  -«^_cotts  -e-j_l 
23  ooste-^s  _1 

29  co-crete_«a_l 
20  MC  mber  1 


Figure  61 .  Map  of  Changes  in  Scenario  1 
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Figure  62.  Scenario  1,  Change  1  (before):  tire_1 


Figure  63.  Scenario  1,  Change  1  (after):  tire_1 
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Figure  64.  Scenario  1 ,  Change  2  (before):  burnt_truck_1 


Figure  65.  Scenario  1 ,  Change  2  (after):  burnt_truck_1 
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Figure  66.  Scenario  1 ,  Change  3  (before):  propane_tank_sales_1 


Figure  67.  Scenario  1 ,  Change  3  (after):  propane_tank_sales_1 
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Figure  68.  Scenario  1,  Change  4  (before):  broken_wall_1 


Figure  69.  Scenario  1 ,  Change  4  (after):  broken_wall_1 
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Figure  70.  Scenario  1 ,  Change  5  (before):  dirt_pile_1 


Figure  7 1 .  Scenario  1 ,  Change  5  (after):  dirt_pile_1 
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Figure  72.  Scenario  1 ,  Change  6  (before):  observer_1 


Figure  73.  Scenario  1 ,  Change  6  (after):  observer_1 
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Figure  74.  Scenario  1,  Change  7  (before):  vid_camera_1 


Figure  75.  Scenario  1 ,  Change  7  (after):  vid_camera_1 
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Figure  76.  Scenario  1,  Change  8  (before):  running_man_1 


Figure  77.  Scenario  1 ,  Change  8  (after):  running_man_1 
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Figure  78.  Scenario  1,  Change  9  (before):  ied_box_1 


Figure  79.  Scenario  1 ,  Change  9  (after):  ied_box_1 
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Figure  80.  Scenario  1,  Change  10  (before):  ied_supply_man_1 


Figure  81.  Scenario  1,  Change  10  (after):  ied_supply_man_1 
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Figure  82.  Scenario  1,  Change  1 1  (before):  odd_drums_1 


Figure  83.  Scenario  1,  Change  1 1  (after):  odd_drums_1 
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Figure  84.  Scenario  1,  Change  12  (before):  vbied_van_1 


Figure  85.  Scenario  1,  Change  12  (after):  vbied_van_1 
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Figure  86.  Scenario  1,  Change  13  (before):  ied_patch_1 
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Figure  88.  Scenario  1,  Change  14  (before):  burkha_girl_1 


Figure  89.  Scenario  1,  Change  14  (after):  burkha_girl_1 
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Figure  90.  Scenario  1,  Change  15  (before):  ruined_house_1 


Figure  91.  Scenario  1,  Change  15  (after):  ruined_house_1 
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Figure  92.  Scenario  1,  Change  16  (before):  broken_wall_2 


Figure  93.  Scenario  1,  Change  16  (after):  broken_wall_2 
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Figure  94.  Scenario  1,  Change  17  (before):  bad_trash_pile_1 


Figure  95.  Scenario  1,  Change  17  (after):  bad_trash_pile_1 
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Figure  96.  Scenario  1,  Change  18  (before):  broken_wall_ied_1 
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Figure  98.  Scenario  1,  Change  19  (before):  broken_guardrail_1 


Figure  99.  Scenario  1,  Change  19  (after):  broken_guardrail_1 
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Figure  100.  Scenario  1,  Change  20  (before):  bad_curb_1 


Figure  101 .  Scenario  1 ,  Change  20  (after):  bad_curb_1 
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Figure  102.  Scenario  1,  Change  21  (before):  reporter_1 


Figure  103.  Scenario  1,  Change  21  (after):  reporter_1 


200 


Figure  104.  Scenario  1,  Change  22  (before):  bad_bag_pile_1 


Figure  105.  Scenario  1,  Change  22  (after):  bad_bag_pile_1 
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Figure  106.  Scenario  1,  Change  23  (before):  bad_barrels_1 


Figure  107.  Scenario  1,  Change  23  (after):  bad_barrels_1 
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Figure  108.  Scenario  1,  Change  24  (before):  bad_rubble_1 


Figure  109.  Scenario  1,  Change  24  (after):  bad_rubble_1 
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Figure  1 10.  Scenario  1 ,  Change  25  (before):  broken_fence_no_ip_1 


Figure  111.  Scenario  1 ,  Change  25  (after):  broken_fence_no_ip_1 
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Figure  112.  Scenario  1,  Change  26  (before):  buried_tank_1 


Figure  113.  Scenario  1,  Change  26  (after):  buried_tank_1 
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Figure  1 14.  Scenario  1,  Change  27  (before):  new_containers_1 


Figure  115.  Scenario  1,  Change  27  (after):  new_containers_1 
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Figure  1 16.  Scenario  1,  Change  28  (before):  poster_wall_1 


Figure  117.  Scenario  1 ,  Change  28  (after):  poster_wall_1 
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Figure  118.  Scenario  1 ,  Change  29  (before):  concreteJecM 


Figure  119.  Scenario  1 ,  Change  29  (after):  concreteJecM 
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Figure  120.  Scenario  1,  Change  30  (before):  bad_mixer_1 


Figure  121.  Scenario  1,  Change  30  (after):  bad_mixer_1 
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B.  SCENARIO  2 


Figure  122.  Scenario  2,  Change  1  (before):  truck_wreck 


Figure  123.  Scenario  2,  Change  1  (after):  truck_wreck 


210 


Figure  124.  Scenario  2,  Change  2  (before):  fridge_no_play 


Figure  125.  Scenario  2,  Change  2  (after):  fridge_no_play 
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Figure  126.  Scenario  2,  Change  3  (before):  suicide_bomber 


Figure  127.  Scenario  2,  Change  3  (after):  suicide_bomber 
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Figure  128.  Scenario  2,  Change  4  (before):  dead_body 


Figure  129.  Scenario  2,  Change  4  (after):  dead_body 
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Figure  130.  Scenario  2,  Change  5  (before):  bombjmaker 


Figure  131 .  Scenario  2,  Change  5  (after):  bombjmaker 
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Figure  132.  Scenario  2,  Change  6  (before):  new_rock 


Figure  133.  Scenario  2,  Change  6  (after):  new_rock 
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Figure  134.  Scenario  2,  Change  7  (before):  new_van 


Figure  135.  Scenario  2,  Change  7  (after):  new_van 
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Figure  136.  Scenario  2,  Change  8  (before):  boarded_up_shop 


Figure  137.  Scenario  2,  Change  8  (after):  boarded_up_shop 
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Figure  138.  Scenario  2,  Change  9  (before):  weird_dudes 
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Figure  140.  Scenario  2,  Change  10  (before):  pressjman 


Figure  141 .  Scenario  2,  Change  10  (after):  pressjman 
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Figure  142.  Scenario  2,  Change  1 1  (before):  bomb_car 


Figure  143.  Scenario  2,  Change  1 1  (after):  bomb_car 
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Figure  144.  Scenario  2,  Change  12  (before):  back_to_back_vans 


Figure  145.  Scenario  2,  Change  12  (after):  back_to_back_vans 
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Figure  146.  Scenario  2,  Change  13  (before):  UN_sacks 


Figure  147.  Scenario  2,  Change  13  (after):  UN_sacks 
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Figure  148.  Scenario  2,  Change  14  (before):  ice_cream_dude 


Figure  149.  Scenario  2,  Change  14  (after):  ice_cream_dude 
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Figure  150.  Scenario  2,  Change  15  (before):  bad_pile 


Figure  151.  Scenario  2,  Change  15  (after):  bad_pile 
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Figure  152.  Scenario  2,  Change  16  (before):  new_boxes_no_phone_booth 


Figure  153.  Scenario  2,  Change  16  (after):  new_boxes_no_phone_booth 
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Figure  154.  Scenario  2,  Change  17  (before):  new_tractor 


Figure  155.  Scenario  2,  Change  17  (after):  new_tractor 
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Figure  156.  Scenario  2,  Change  18  (before):  nobody_ia_cp 


Figure  157.  Scenario  2,  Change  18  (after):  nobody_ia_cp 


227 


Figure  158.  Scenario  2,  Change  19  (before):  new_car_open_street 


228 


Figure  160.  Scenario  2,  Change  20  (before):  broken_wall 


Figure  161 .  Scenario  2,  Change  20  (after):  broken_wall 
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Figure  162.  Scenario  2,  Change  21  (before):  turned_truck 


Figure  163.  Scenario  2,  Change  21  (after):  turned_truck 
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Figure  164.  Scenario  2,  Change  22  (before):  jingle_truck 


Figure  165.  Scenario  2,  Change  22  (after):  jingle_truck 
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Figure  166.  Scenario  2,  Change  23  (before):  new_rubble_no_kids 


Figure  167.  Scenario  2,  Change  23  (after):  new_rubble_no_kids 
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Figure  168.  Scenario  2,  Change  24  (before):  wall_truck 


Figure  169.  Scenario  2,  Change  24  (after):  wall_truck 
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Figure  170.  Scenario  2,  Change  25  (before):  ip_suicide_bomber 


Figure  171 .  Scenario  2,  Change  25  (after):  ip_suicide_bomber 
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Figure  172.  Scenario  2,  Change  26  (before):  propane_tanks 


Figure  173.  Scenario  2,  Change  26  (after):  propane_tanks 
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Figure  174.  Scenario  2,  Change  27  (before):  bad_cart 


Figure  175.  Scenario  2,  Change  27  (after):  bad_cart 
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Figure  176.  Scenario  2,  Change  28  (before):  new_cars 


Figure  177.  Scenario  2,  Change  28  (after):  new_cars 


Figure  178.  Scenario  2,  Change  29  (before):  van_bomb 


Figure  179.  Scenario  2,  Change  29  (after):  van_bomb 
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Figure  180.  Scenario  2,  Change  30  (before):  new_barrels 


Figure  181 .  Scenario  2,  Change  30  (after):  new_barrels 
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APPENDIX  L.  POTENTIAL  TARGET  (CHANGE)  LOCATIONS 


A.  OVERVIEW  MAP 


Figure  182.  Overview  Map  of  VBS2™  scene  locations 
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B.  LOCATIONS  FOR  POTENTIAL  CHANGES  IN  A  SCENE 


In  the  following  112-screen  captures,  red  crosses  mark  the  nearly  5000 
locations  where  a  change  could  potentially  occur  in  the  environment.  The  yellow 
crosses  represent  locations  already  labeled  with  a  red  cross  in  another  figure. 


Figure  183.  Scene  1,  left  side  of  vehicle,  Route  Blue,  19  potential  changes 
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Figure  184.  Scene  2,  left  side  of  vehicle,  Route  Blue,  31  potential  changes 


Figure  185.  Scene  3,  left  side  of  vehicle,  Route  Blue,  64  potential  changes 
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Figure  186.  Scene  4,  left  side  of  vehicle,  Route  Blue,  32  potential  changes 


Figure  187.  Scene  5,  left  side  of  vehicle,  Route  Blue,  33  potential  changes 
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Figure  188.  Scene  6,  left  side  of  vehicle,  Route  Blue,  50  potential  changes 


Figure  189.  Scene  7,  left  side  of  vehicle,  Route  Blue,  37  potential  changes 
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Figure  190.  Scene  8,  left  side  of  vehicle,  Route  Blue,  15  potential  changes 


Figure  191.  Scene  9,  left  side  of  vehicle,  Route  Blue,  21  potential  changes 
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Figure  192.  Scene  10,  left  side  of  vehicle,  Route  Blue,  31  potential  changes 


Figure  1 93.  Scene  1 1 ,  left  side  of  vehicle,  Route  Blue,  23  potential  changes 
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Figure  194.  Scene  12,  left  side  of  vehicle,  Route  Red,  39  potential  changes 


Figure  195.  Scene  13,  right  side  of  vehicle,  Route  Blue,  17  potential  changes 
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Figure  196.  Scene  14,  right  side  of  vehicle,  Route  Blue,  49  potential  changes 


Figure  197.  Scene  15,  right  side  of  vehicle,  Route  Blue,  49  potential  changes 
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Figure  198.  Scene  16,  right  side  of  vehicle,  Route  Blue,  66  potential  changes 


Figure  199.  Scene  17,  right  side  of  vehicle,  Route  Blue,  98  potential  changes 


250 


Figure  200.  Scene  1 8,  right  side  of  vehicle,  Route  Blue,  44  potential  changes 


Figure  201 .  Scene  1 9,  right  side  of  vehicle,  Route  Blue,  93  potential  changes 


251 


Figure  202.  Scene  20,  right  side  of  vehicle,  Route  Blue,  29  potential  changes 


Figure  203.  Scene  21 ,  right  side  of  vehicle,  Route  Blue,  27  potential  changes 
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Figure  204.  Scene  22,  right  side  of  vehicle,  Route  Blue,  35  potential  changes 


Figure  205.  Scene  23,  right  of  vehicle,  Route  Blue,  23  potential  changes 
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Figure  206.  Scene  24,  right  of  vehicle,  Route  Blue,  13  potential  changes 


Figure  207.  Scene  25,  left  of  vehicle,  Route  Red,  31  potential  changes 
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Figure  208.  Scene  26,  left  of  vehicle,  Route  Red,  31  potential  changes 


Figure  209.  Scene  27,  left  of  vehicle,  Route  Red,  46  potential  changes 
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Figure  210.  Scene  28,  left  of  vehicle,  Route  Red,  61  potential  changes 


Figure  21 1 .  Scene  29,  left  of  vehicle,  Route  Red,  54  potential  changes 
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Figure  212.  Scene  30,  left  of  vehicle,  Route  Red,  56  potential  changes 


Figure  213.  Scene  31 ,  left  side  of  vehicle,  Route  Red,  63  potential  changes 
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Figure  214.  Scene  32,  left  side  of  vehicle,  Route  Red,  35  potential  changes 


Figure  215.  Scene  33,  left  side  of  vehicle,  Route  Red,  50  potential  changes 
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Figure  216.  Scene  34,  left  side  of  vehicle,  Route  Red,  36  potential  changes 


Figure  217.  Scene  35,  left  side  of  vehicle,  Route  Red,  37  potential  changes 
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Figure  218.  Scene  36,  left  side  of  vehicle,  Route  Red,  47  potential  changes 


Figure  219.  Scene  37,  left  side  of  vehicle,  Route  Red,  53  potential  changes 
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Figure  220.  Scene  38,  left  side  of  vehicle,  Route  Red,  14  potential  changes 


Figure  221 .  Scene  39,  left  side  of  vehicle,  Route  Green,  22  potential  changes 
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Figure  222.  Scene  40,  right  side  of  vehicle,  Route  Red,  46  potential  changes 


Figure  223.  Scene  41 ,  right  side  of  vehicle,  Route  Red,  58  potential  changes 
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Figure  224.  Scene  42,  right  side  of  vehicle,  Route  Red,  55  potential  changes 


Figure  225.  Scene  43,  right  of  vehicle,  Route  Red,  29  potential  changes 
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Figure  226.  Scene  44,  right  of  vehicle,  Route  Red,  63  potential  changes 


Figure  227 .  Scene  45,  right  side  of  vehicle,  Route  Red,  62  potential  changes 
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Figure  228.  Scene  46,  right  side  of  vehicle,  Route  Red,  35  potential  changes 


Figure  229.  Scene  47,  right  side  of  vehicle,  Route  Red,  41  potential  changes 
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Figure  230.  Scene  48,  right  side  of  vehicle,  Route  Red,  28  potential  changes 


Figure  231 .  Scene  49,  right  side  of  vehicle,  Route  Red,  46  potential  changes 
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Figure  232.  Scene  50,  right  side  of  vehicle,  Route  Red,  52  potential  changes 


Figure  233.  Scene  51 ,  right  side  of  vehicle,  Route  Red,  37  potential  changes 


267 


Figure  234.  Scene  52,  right  side  of  vehicle,  Route  Red,  36  potential  changes 


Figure  235.  Scene  53,  right  side  of  vehicle,  Route  Red,  52  potential  changes 
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Figure  236.  Scene  54,  right  side  of  vehicle,  Route  Red,  38  potential  changes 


Figure  237.  Scene  55,  left  side  of  vehicle,  Route  Green,  25  potential  changes 
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Figure  238.  Scene  56,  left  side  of  vehicle,  Route  Green,  20  potential  changes 


Figure  239.  Scene  57,  left  side  of  vehicle,  Route  Green,  28  potential  changes 
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Figure  240.  Scene  58,  left  side  of  vehicle,  Route  Green,  18  potential  changes 


Figure  241 .  Scene  59,  left  side  of  vehicle,  Route  Green,  27  potential  changes 
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Figure  242.  Scene  60,  left  side  of  vehicle,  Route  Green,  18  potential  changes 


Figure  243.  Scene  61 ,  left  side  of  vehicle,  Route  Green,  14  potential  changes 


272 


Figure  244.  Scene  62,  left  side  of  vehicle,  Route  Green,  14  potential  changes 


Figure  245.  Scene  63,  right  side  of  vehicle,  Route  Green,  64  potential  changes 
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Figure  246.  Scene  64,  right  side  of  vehicle,  Route  Green,  37  potential  changes 


Figure  247.  Scene  65,  right  side  of  vehicle,  Route  Green,  51  potential  changes 
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Figure  248.  Scene  66,  right  side  of  vehicle,  Route  Green,  52  potential  changes 


I 
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Figure  249.  Scene  67,  right  side  of  vehicle,  Route  Green,  48  potential  changes 
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Figure  250.  Scene  68,  right  side  of  vehicle,  Route  Green,  51  potential  changes 


Figure  251 .  Scene  69,  right  side  of  vehicle,  Route  Green,  56  potential  changes 
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Figure  252.  Scene  70,  right  side  of  vehicle,  Route  Green,  89  potential  changes 


Figure  253.  Scene  71 ,  left  side  of  vehicle,  Route  Purple,  44  potential  changes 
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Figure  254.  Scene  72,  left  side  of  vehicle,  Route  Purple,  45  potential  changes 


Figure  255.  Scene  73,  left  side  of  vehicle,  Route  Purple,  35  potential  changes 
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Figure  256.  Scene  74,  left  side  of  vehicle,  Route  Purple,  45  potential  changes 


Figure  257.  Scene  75,  left  side  of  vehicle,  Route  Purple,  52  potential  changes 
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Figure  258.  Scene  76,  left  side  of  vehicle,  Route  Purple,  20  potential  changes 


Figure  259.  Scene  77,  left  side  of  vehicle,  Route  Purple,  43  potential  changes 
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Figure  260.  Scene  78,  left  side  of  vehicle,  Route  Purple,  70  potential  changes 


Figure  261 .  Scene  79,  left  side  of  vehicle,  Route  Purple,  21  potential  changes 
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Figure  262.  Scene  80,  left  side  of  vehicle,  Route  Purple,  21  potential  changes 


Figure  263.  Scene  81 ,  left  side  of  vehicle,  Route  Purple,  22  potential  changes 
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Figure  264.  Scene  82,  left  side  of  vehicle,  Route  Purple,  30  potential  changes 


Figure  265.  Scene  83,  left  side  of  vehicle,  Route  Purple,  41  potential  changes 
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Figure  266.  Scene  84,  left  side  of  vehicle,  Route  Purple,  66  potential  changes 
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Figure  268.  Scene  86,  left  side  of  vehicle,  Route  Purple,  53  potential  changes 


Figure  269.  Scene  87,  left  side  of  vehicle,  Route  Purple,  34  potential  changes 
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Figure  270.  Scene  88,  left  side  of  vehicle,  Route  Purple,  29  potential  changes 


Figure  271 .  Scene  89,  left  side  of  vehicle,  Route  Purple,  29  potential  changes 
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Figure  272.  Scene  90,  left  side  of  vehicle,  Route  Purple,  69  potential  changes 


Figure  273.  Scene  91 ,  left  side  of  vehicle,  Route  Purple,  37  potential  changes 
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Figure  274.  Scene  92,  right  side  of  vehicle,  Route  Purple,  44  potential  changes 


Figure  275.  Scene  93,  right  side  of  vehicle,  Route  Purple,  67  potential  changes 
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Figure  276.  Scene  94,  right  side  of  vehicle,  Route  Purple,  32  potential  changes 


Figure  277.  Scene  95,  right  side  of  vehicle,  Route  Purple,  27  potential  changes 
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Figure  278.  Scene  96,  right  side  of  vehicle,  Route  Purple,  45  potential  changes 


Figure  279.  Scene  97,  right  side  of  vehicle,  Route  Purple,  31  potential  changes 
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Figure  280.  Scene  98,  right  side  of  vehicle,  Route  Purple,  41  potential  changes 


Figure  281 .  Scene  99,  right  side  of  vehicle,  Route  Purple,  54  potential  changes 
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Figure  282.  Scene  100,  right  of  vehicle,  Route  Purple,  30  potential  changes 


Figure  283.  Scene  101,  right  of  vehicle,  Route  Purple,  61  potential  changes 
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Figure  284.  Scene  102,  right  of  vehicle,  Route  Purple,  147  potential  changes 


Figure  285.  Scene  103,  right  of  vehicle,  Route  Purple,  154  potential  changes 
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Figure  286.  Scene  104,  right  of  vehicle,  Route  Purple,  157  potential  changes 


Figure  287.  Scene  105,  right  of  vehicle,  Route  Purple,  19  potential  changes 
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Figure  288.  Scene  106,  right  of  vehicle,  Route  Purple,  23  potential  changes 


Figure  289.  Scene  107,  right  of  vehicle,  Route  Purple,  37  potential  changes 
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Figure  290.  Scene  108,  right  of  vehicle,  Route  Purple,  72  potential  changes 


Figure  291.  Scene  109,  right  of  vehicle,  Route  Purple,  76  potential  changes 
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Figure  292.  Scene  110,  right  of  vehicle,  Route  Purple,  42  potential  changes 


Figure  293.  Scene  111,  right  of  vehicle,  Route  Purple,  29  potential  changes 
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Figure  294.  Scene  112,  front  of  vehicle,  Route  Purple,  13  potential  changes 
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